redis performance problem

[wociscz]

Hi, i just recognised the performance problem. After a few weeks my redis db is ~400MB on disk.
rspamd process sending periodically these commands to redis (allmost every second)
KEYS BAYES_SPAM* (which gets allways 1 key BAYES_SPAM)
KEYS BAYES_HAM* (which gets allways 1 key BAYES_HAM)
and HGET, HLEN variants.

It seems to be for statistics of how many learns we have or something like that.

In my situation these commands lasts ~10sec to get them done. And performance of redis is going to the hell. I have hundreds of connect timeouts to redis and rspamd is allmost useles in this situation.

In the redis doc, there is:

IMPORTANT NOTE: a VERY common source of latency generated by the execution of slow commands is the use of the KEYS command in production environments. KEYS, as documented in the Redis documentation, should only be used for debugging purposes. Since Redis 2.8 a new commands were introduced in order to iterate the key space and other large collections incrementally, please check the SCAN, SSCAN, HSCAN and ZSCAN commands for more information.

Is it possible to turn off these blocking commands, or are they important for some internals?

When i clean redis db, it works fast, so with bigger redis db these commands is slower and slower to the point where everything stops...

actual rspamc stat

Results for command: stat (0.015 seconds)
Messages scanned: 15979385
Messages with action reject: 10946421, 68.50%
Messages with action soft reject: 60936, 0.38%
Messages with action rewrite subject: 958341, 5.99%
Messages with action add header: 1322576, 8.27%
Messages with action greylist: 691139, 4.32%
Messages with action no action: 1999972, 12.51%
Messages treated as spam: 13288274, 83.15%
Messages treated as ham: 2691111, 16.84%
Messages learned: 10851525
Connections count: 13476
Control connections count: 4
Pools allocated: 13501
Pools freed: 13348
Bytes allocated: 3.96M
Memory chunks allocated: 489
Shared chunks allocated: 10
Chunks freed: 0
Oversized chunks: 3
Fuzzy hashes in storage "rspamd.com": 420130
Fuzzy hashes stored: 420130
Statfile: BAYES_SPAM type: redis; length: 67.11M; free blocks: 0; total blocks: 1.76M; free: 0.00%; learned: 80594; users: 1; languages: 0
Statfile: BAYES_HAM type: redis; length: 151.28M; free blocks: 0; total blocks: 3.98M; free: 0.00%; learned: 16460; users: 1; languages: 0
Total learns: 0

redis infos:

Keyspace

db0:keys=2607839,expires=2586332,avg_ttl=61159389

Memory

used_memory:910873168
used_memory_human:868.68M
used_memory_rss:1006358528
used_memory_rss_human:959.74M
used_memory_peak:979017744
used_memory_peak_human:933.66M
total_system_memory:25327726592
total_system_memory_human:23.59G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.10
mem_allocator:jemalloc-4.0.3

Server

redis_version:3.2.2

rspamd version
1.3.0-0git1738c58447~jessie
(but same behaviour on the newest 1.3.2*)

Regards,
Jan

[vstakhov]

That should be fixed now, thank you for the report.