used_cpu_user keep increasing without traffic

[smartnews-weitao]

I setup my redis cluster on top of kubernetes. And the kuberneters's nodes are AWS's EC2. Recently i have been found that one of my cluster's used_cpu_user has been keep increasing. I used redis-cli's monitor feature check ongoing traffic, but there are no traffic except client's ping. My used_cpu_user is around 20582 without any traffic. Here is my redis cluster's config: Please take a look and give me some clue.

cluster-enabled yes
cluster-require-full-coverage no
cluster-node-timeout 15000
cluster-replica-validity-factor 0
cluster-config-file /data/nodes.conf
cluster-migration-barrier 1
cluster-allow-reads-when-down yes
appendonly no
no-appendfsync-on-rewrite yes
auto-aof-rewrite-min-size 1024mb
save 300 100
protected-mode no
maxmemory 30gb
maxmemory-policy allkeys-lru
io-threads 2
logfile "redis.log"
repl-backlog-size 512mb
activedefrag yes
active-defrag-threshold-lower 5
active-defrag-threshold-upper 15

[smartnews-weitao]

Seems related to activedefrag

[oranagra]

@smartnews-weitao please state which version of redis are you using, and maybe also post two snapshots of the INFO ALL output taken one minute apart so we can see the changes in statistics and metrics.

You can try disabling activedefrag and see if it solves the CPU utilization problem, then when we know that, we can look into the fragmentation, figure out where it comes from and if/why active defrag can't fix it.

Maybe the 5% threshold is too low, and the existing >5% fragmentation is located somewhere where the defragger isn't able to fix it, in which case it'll keep working forever and never get anything done. if that's true, increasing the threshold to above the constant non-defraggable value is the right thing to do (that's why the default is 10%, a value that's too low is risky).