-
Notifications
You must be signed in to change notification settings - Fork 149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The cache has been hit, but the time taken to read the key will exceed 10ms #89
Comments
This will be better when I restart the service. |
@xiluoxi, thank you for reporting this. I will look into that as soon as possible. |
Hi @xiluoxi, please try the new v0.0.74. The memory leak in the LRU cache should be fixed. |
@rueian After the service runs for a period of time, the memory usage will still rise. |
In the test, the memory increased abnormally, from 4.1% to 21%.Next, the performance of rueidis improved |
Hi @xiluoxi, just want to clarify. Do you expect that it keeps using 4.1% of memory? How much time did it take to occupying 21% of memory?
Do you mean that though there is still memory leak issue, but the latency issue is solved? |
The latency issue still exists。 |
At the same time, a new problem has emerged. In a highly concurrent write scenario,the memory usage will increase rapidly when redis fails or processes slowly. |
In a highly concurrent read scenario,The latency > 100ms |
There are two rueidis clients connect to different redis servers in my service, one for reading and one for writing.I'm not sure they can influence each other. |
They should not affect each other. What is the relationship between these two Redis servers? Are they Redis Cluster? BTW, Are you using |
No Redis cluster, and they are on two different servers. using |
Hi @xiluoxi,
This may be caused by the fact that, currently, the command builder does not reuse the command buffer of previously failed commands due to some racing problems. This may take some time to improve.
I have done some tests on Google Cloud but I am still not able to simulate your situation. I created two instances in the same zone of Google Cloud, their spec:
The first machine was running Redis 7.0.4 + Prometheus + Grafana package main
import (
"context"
"fmt"
"math/rand"
"net/http"
"strconv"
"time"
"github.com/go-redis/redis/v9"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/rueian/rueidis"
)
func prepData(n int) []string {
data := make([]string, n)
for i := range data {
data[i] = strconv.Itoa(i)
}
rand.Shuffle(len(data), func(i, j int) { data[i], data[j] = data[j], data[i] })
return data
}
const (
keyCount = 1000000
readers = 8
writers = 2
useGoRedis = false // please change it
cacheSize = 512 * (1 << 20) // 512 MB
addr = "10.140.0.52:6379" // please change it
)
func main() {
rand.Seed(time.Now().UnixNano())
bucket := []float64{250, 500, 750, 1000, 2500, 5000, 7500, 10000, 25000, 50000, 75000, 100000, 250000, 500000, 750000, 1000000}
wl := promauto.NewHistogram(prometheus.HistogramOpts{Name: "micro_write_latency", Buckets: bucket})
rl := promauto.NewHistogram(prometheus.HistogramOpts{Name: "micro_read_latency", Buckets: bucket})
go func() {
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe(":2112", nil)
}()
rc, err := rueidis.NewClient(rueidis.ClientOption{
InitAddress: []string{addr},
CacheSizeEachConn: cacheSize,
})
if err != nil {
panic(err)
}
gc := redis.NewUniversalClient(&redis.UniversalOptions{
Addrs: []string{addr},
})
ctx := context.Background()
goredisWrite := func(key, data string) error {
return gc.Set(ctx, key, data, 0).Err()
}
goredisRead := func(key string) error {
return gc.Get(ctx, key).Err()
}
rueidisWrite := func(key, data string) error {
return rc.Do(ctx, rc.B().Set().Key(key).Value(data).Build()).Error()
}
rueidisCache := func(key string) error {
return rc.DoCache(ctx, rc.B().Get().Key(key).Cache(), time.Hour).Error()
}
var wfn func(key, data string) error
var rfn func(key string) error
if useGoRedis {
wfn = goredisWrite
rfn = goredisRead
} else {
wfn = rueidisWrite
rfn = rueidisCache
}
writeFn := func(keys, data []string) {
for i, k := range keys {
ts := time.Now()
err := wfn(k, data[i])
wl.Observe(float64(time.Since(ts).Microseconds()))
if err != nil {
panic(err)
}
}
}
readFn := func(keys []string) {
for _, k := range keys {
ts := time.Now()
err := rfn(k)
rl.Observe(float64(time.Since(ts).Microseconds()))
if err != nil {
panic(err)
}
}
}
{
keys := prepData(keyCount)
data := prepData(keyCount)
commands := make(rueidis.Commands, len(keys))
for i := range commands {
commands[i] = rc.B().Set().Key(keys[i]).Value(data[i]).Build()
}
ts := time.Now()
for _, resp := range rc.DoMulti(ctx, commands...) {
if err := resp.Error(); err != nil {
panic(err)
}
}
fmt.Println("ready", time.Since(ts))
}
if useGoRedis {
rc.Close()
} else {
gc.Close()
}
for i := 0; i < writers; i++ {
go func() {
keys := prepData(keyCount)
data := prepData(keyCount)
for {
writeFn(keys, data)
}
}()
}
for i := 0; i < readers; i++ {
go func() {
keys := prepData(keyCount)
for {
readFn(keys)
}
}()
}
time.Sleep(time.Hour)
} This program records metrics of 8 concurrent readers and 2 concurrent writers that keep reading and writing 1000000 keys. The result of rueidis v0.0.75 with an additional 512MB client-side cache: While indeed rueidis used more memory for client-side caching, it achieved 14x read throughput over goredis (887874/61978) in this case with P99 latencies <0.5ms and no memory leak. Would you mind sharing more details about your machine/network spec and traffic patterns like concurrency, read/write ratio, cache-hit ratio as well as avg key/value size? So that I can help you and find the causes of your problem. |
You can try this case, read and write the same key with high concurrency, mainly reading. |
Hi @xiluoxi, the previous simulation I posted is reading and writing the set of keys with high concurrency, mainly reading. |
Hi @xiluoxi, There are 3 new fields of the Increasing the You can try to increase or decrease them to see how they will affect performance and find better values for your case. This is the result of the same code and same machines of the previous simulation but with the v0.0.76: |
Thanks,I will try. |
Usually, this occurs after the service has been running for a long time.I guess it might be after the cache is full.
The text was updated successfully, but these errors were encountered: