redis cluster

[celevra]

Classification (Please choose one option):

• [] Crash/Hang/Data loss
• WebUI/Usability
• Serious bug
• Other bug
• Feature
• Enhancement
• not sure

Reproducibility (Please choose one option):

• Always
• Sometimes
• Rarely
• Unable
• I didn’t try
• Not applicable

Rspamd version:

1.6.5-1~trusty

Operation system, CPU, memory and environment:

Ubuntu 14.04.5 LTS

Description (Please provide a descriptive summary of the issue):

changed to redis cluster, now got MOVED errors

Compile errors (if any):

Steps to reproduce:

Expected results:

Actual results:

Debugging information (see details here):

Configuration:

Additional information:

Hi,

at first, thank you for this great piece of software!
I've run it with a local redis instance, now i've transformed this redis instance into a redis 6 node cluster.
After this, i see a lots of errors like this:

2017-10-26 13:03:30 #4380(rspamd_proxy) <5a0b8d>; lua; antivirus.lua:293: Got error checking cache: MOVED 15011 2a01:4f8:172:xxx:4000::13:6379
2017-10-26 13:03:30 #4380(rspamd_proxy) <5a0b8d>; lua; antivirus.lua:324: failed to save virus cache for OK -> "rs_cla3f79fc65766702c25f169a8ec32be2e": MOVED 15011 2a01:4f8:172:xxx:4000::13:6379
2017-10-26 13:03:30 #4380(rspamd_proxy) <5a0b8d>; proxy; rspamd_redis_connected: bad learned type for BAYES_SPAM: error, nil expected
2017-10-26 13:03:30 #4380(rspamd_proxy) <5a0b8d>; proxy; rspamd_redis_processed: got invalid reply from redis: error, array expected

2017-10-26 13:02:39 #10816(rspamd_proxy) <74af04>; lua; greylist.lua:225: got error while getting greylisting keys: CROSSSLOT Keys in request don't hash to the same slot
2017-10-26 13:02:39 #10816(rspamd_proxy) <74af04>; lua; antivirus.lua:293: Got error checking cache: MOVED 4120 2a01:4f8:xxx:32a7::13:6379
2017-10-26 13:02:39 #10816(rspamd_proxy) <74af04>; lua; ip_score.lua:230: Redis error: MOVED 12671 2a01:4f8:172:xxx:4000::13:6379
2017-10-26 13:02:39 #10816(rspamd_proxy) <74af04>; lua; antivirus.lua:324: failed to save virus cache for OK -> "rs_cle0c3b1054d74e95fc7a969697cefc65c": MOVED 4120 2a01:4f8:xxx:32a7::13:6379

2017-10-26 13:03:49 #10816(rspamd_proxy) <o7ddn4>; lua; fann_redis.lua:807: cannot get FANNS list from redis: MOVED 1814 2a01:4f8:xxx:32a7::13:6379
2017-10-26 13:03:49 #10818(controller) <o7ddn4>; lua; fann_redis.lua:807: cannot get FANNS list from redis: MOVED 1814 2a01:4f8:xxx:32a7::13:6379
2017-10-26 13:04:49 #10816(rspamd_proxy) <o7ddn4>; lua; fann_redis.lua:807: cannot get FANNS list from redis: MOVED 1814 2a01:4f8:xxx:32a7::13:6379

/etc/rspam.d/local.d/redis.conf does not handel ipv6 adresses, so i've used hostnames and mapped them with /etc/hosts

servers = "redisM01,redisM02,redisM03,redisS01,redisS02,redisS03"

so i have to clear any file caches or is it a real problem?

thank you for your help

[vstakhov]

Redis cluster is not supported and there are no plans to add support for it. https://rspamd.com/doc/quickstart.html#using-rspamd-in-large-email-systems

[celevra]

hm, i don't get why :-)
i've got a loadbalancer and behind it mx01 and mx02, both run rspamd.

if mx01 dies, the loadbalancer reroutes all traffic to mx02, and i want rspamd to be fully usable.
But if i only have a master-slave redis, mx02 can't write new keys, couldn't it?

but i need "synchronised" rspamd because of for example graylisting, mx01 greylists an email, and on the second connect it goes to mx02, so i don't want rspamd to graylist it again.

do i get anything wrong?

[greem]

I've recently made a change which will be in the next release of Exim, the MTA, to 'support' (for some value of the word) redis cluster.

In short: exim is provided with the addresses of all redis instances in the cluster (masters and slaves) in a list. In the event that a write or read returns 'MOVED ', exim regards that response as a deferral rather than a failure, and moves on to the next instance in the list. Assuming the MOVED response is referring to an instance in exim's list - if it isn't, exim is misconfigured - then after N deferrals the write or read will be made to the redis instance which contains that key.

It may not be very efficient, especially in very large systems with many redis instances, but I have it working successfully on an email platform which runs at between 250k and a million messages per day, with an 8 node redis cluster (4 masters, 4 slaves).

I'll have a look and see if the same thing could be done with rspamd, but it'll be my first venture in lua...

[qoke]

Give this a try: https://github.com/eleme/corvus

We are using this to front a 12 node redis cluster with a "classic redis" API.

[qoke]

Give this a try: https://github.com/eleme/corvus

We are using this to front a 12 node redis cluster with a "classic redis" API.

One (important) thing to note is that (at present) neither corvus, codis or twemproxy support the SCRIPT LOAD command, which rspamd uses. This will eventually either stop you using features of rspamd or stop you using redis cluster, and revert to a master+slave redis setup.

Using a master/slave setup has problems with scaling the writes, and having to deal with "realtime" failover (that is not very realtime) in a very convoluted/hacky manner.

[vstakhov]

There are no issues with scaling when using master/slave replication for non-volatile data and sharding for volatile data. Rspamd also supports Redis Sentinel for convenience. Again, there are no plans to support Redis Cluster, as it does not resolve anything but adds extra complexity especially when using pool of connections.

[bra-fsn]

BTW on its page rspamd now recommends KeyDB.