Cluster - Moved/Ask Redirection Not Working Properly.

[Tbone542]

Expected behaviour

Running a cluster and using the slot cache. When I migrate keys I would expect the slot cache to update if it receives an ASK or MOVE.

Actual behaviour

Upon requesting the data from a key on a migrated slot, the cache does not seem to update and continually requests keys from the wrong node. If I restart apache the slot cache is refreshed and works fine.

Update: It appears ASK and MOVE redirects are not being handled properly.

I'm seeing this behaviour on

• OS: linux
• Redis: 5.05
• PHP: 7.1
• phpredis: 5.0.2 and 5.1.1

Steps to reproduce, backtrace or example script

I've checked

• [ x ] There is no similar issue from other users
• [ x ] Issue isn't fixed in develop branch

[michael-grunder]

Thanks for the report. I do have logic to invalidate the cache in the event we receive any MOVE or ASK response, but perhaps it's broken.

Hopefully I can replicate the problem.

[Tbone542]

Thank you Michael. I tried to replicate locally and was unsuccessful. The issue became apparent when I performed a small test migration on production. Maybe it has to do with the number of connected clients? It was difficult to emulate locally having hundreds of simultaneous clients.

[Tbone542]

Turns out the boxes with the issue had 5.0.2 not 5.1.1. Could this be part of the problem?

I didn't see anything in the change log that seemed pertinent.

Thanks again,
Ted

[michael-grunder]

There were a couple of changes to the caching code between 5.0.2 and 5.1.1 although I don't recall if they would've been likely to prevent cache invalidation.

This is good to know though, as I'll make sure to check if invalidation is broken in 5.0.2 and not in 5.1.1. Might get lucky and make investigation simpler! 😄

[Tbone542]

Thanks again. After another try with migrating a few slots, it appeared as if the issue arose on only some of the httpd processes and not all. Does that make any sense? i.e. Most requests had no issues finding the right slots, however a few would have problems contacting the cluster.

[michael-grunder]

Sure that makes sense. I assume it's a cache invalidation problem that only happens sometimes, which honestly means it might be trickier to track down.

It's been a while since I've used apache. Are you using prefork or workers (in case I need to use a VM to replicate the problem).

Also a uname -a might end up being helpful.

[Tbone542]

It is on an Amazon Linux distribution, using prefork.

Is the cluster cache shared among children?

When the call fails, it is returning a (boolean) false.
i.e.
return $this->redis->hgetall('somekey');

Thanks again for your help.

[michael-grunder]

Is the cluster cache shared among children?

No I think each fork will have its own persistent_list hash table where we store cached slot information and any persistent connections.

Can you provide the output from phpinfo() so I can set up the VM with the same settings?

[Tbone542]

Which sections do you need?

PHP Version => 7.0.33

redis

Redis Support => enabled
Redis Version => 5.0.2
Available serializers => php, json

Directive => Local Value => Master Value
redis.arrays.algorithm => no value => no value
redis.arrays.auth => no value => no value
redis.arrays.autorehash => 0 => 0
redis.arrays.connecttimeout => 0 => 0
redis.arrays.consistent => 0 => 0
redis.arrays.distributor => no value => no value
redis.arrays.functions => no value => no value
redis.arrays.hosts => no value => no value
redis.arrays.index => 0 => 0
redis.arrays.lazyconnect => 0 => 0
redis.arrays.names => no value => no value
redis.arrays.pconnect => 0 => 0
redis.arrays.previous => no value => no value
redis.arrays.readtimeout => 0 => 0
redis.arrays.retryinterval => 0 => 0
redis.clusters.auth => no value => no value
redis.clusters.cache_slots => 1 => 1
redis.clusters.persistent => 0 => 0
redis.clusters.read_timeout => 0 => 0
redis.clusters.seeds => no value => no value
redis.clusters.timeout => 0 => 0
redis.pconnect.connection_limit => 0 => 0
redis.pconnect.pooling_enabled => 1 => 1
redis.session.lock_expire => 0 => 0
redis.session.lock_retries => 10 => 10
redis.session.lock_wait_time => 2000 => 2000
redis.session.locking_enabled => 0 => 0

[michael-grunder]

That should work. Last question, are you using persistent connections with RedisCluster?

// e.g. passing `true` to the persistent argument
$obj_cluster = new RedisCluster(NULL, ["host:7000", "host:7001"], 1.5, 1.5, true);

[Tbone542]

Yes, persistent connections are being used.

[Tbone542]

So, I did a bit of testing and was able to produce a similar result locally.

I did it using jMeter and a local VM. It was tried with both 5.0.2 and 5.1.1 and each yielded similar results, with and without slot caching enabled. Most of my trials involved 50 -100 http threads and most often 100% of the threads would return a (boolean)false during the slot migration of a particular key.

What I found is in almost every case, that the time during which a slot is being migrated, I am receiving a (boolean)false, when trying to retrieve a key from said slot.

Below is a sample. It shows the Redis monitor output from the original node, then the output from my clients during the migration and then the Redis monitor output from the receiving node, once it has migrated completely. Noticed the microtime of the failures seem to align with the timestamps in the Redis monitor output, during migration.

All of the failures seem to occur after the receiving node gets the RESTORE-ASKING command.

After reviewing the code, it appears there are facilities to deal with ASK redirection, however I am not seeing it work in practice.

Hopefully this information is of some use.

[Node 7001] 1578593676.648111 [0 192.168.56.101:57862] "HGETALL" "mykey"
[Node 7001] 1578593676.648115 [0 192.168.56.101:58466] "HGETALL" "mykey"
[Node 7001] 1578593676.648119 [0 192.168.56.101:58012] "HGETALL" "mykey"
[Node 7001] 1578593676.648122 [0 192.168.56.101:57820] "HGETALL" "mykey"
[Node 7001] 1578593676.648126 [0 192.168.56.101:57872] "HGETALL" "mykey"
[Node 7001] 1578593676.648132 [0 192.168.56.101:58422] "HGETALL" "mykey"
[Node 7001] 1578593676.648136 [0 192.168.56.101:58054] "HGETALL" "mykey"
[Node 7001] 1578593676.648141 [0 192.168.56.101:58136] "MIGRATE" "192.168.56.101" "7002" "" "0" "60000" "KEYS" "mykey"
[Node 7002] 1578593676.648237 [0 192.168.56.101:51574] "RESTORE-ASKING" "mykey" "0" "data payload"

######## Failures Start

0.67937000 1578593676-FAIL...
bool(false)
0.68299400 1578593676-FAIL...
bool(false)

0.68035300 1578593676-FAIL...
bool(false)
0.68348600 1578593676-FAIL...
bool(false)

0.67933800 1578593676-FAIL...
bool(false)
0.68297400 1578593676-FAIL...
bool(false)

######## Failures End

[Node 7002] 1578593676.684348 [0 192.168.56.101:51186] "HGETALL" "mykey"
[Node 7002] 1578593676.684426 [0 192.168.56.101:51210] "HGETALL" "mykey"
[Node 7002] 1578593676.684475 [0 192.168.56.101:51142] "HGETALL" "mykey"
[Node 7002] 1578593676.684564 [0 192.168.56.101:51186] "HGETALL" "mykey"
[Node 7002] 1578593676.684570 [0 192.168.56.101:51210] "HGETALL" "mykey"
[Node 7002] 1578593676.684574 [0 192.168.56.101:51142] "HGETALL" "mykey"
[Node 7002] 1578593676.684579 [0 192.168.56.101:51070] "HGETALL" "mykey"
[Node 7002] 1578593676.684583 [0 192.168.56.101:51636] "HGETALL" "mykey"
[Node 7002] 1578593676.684589 [0 192.168.56.101:51760] "HGETALL" "mykey"
[Node 7002] 1578593676.684593 [0 192.168.56.101:51042] "HGETALL" "mykey"
[Node 7002] 1578593676.684598 [0 192.168.56.101:51762] "HGETALL" "mykey"

[michael-grunder]

This is really useful, thanks!

So maybe the problem is actually just that the ASKING redirection logic is broken. I'm not sure it's been changed since I first wrote it, but a bug could have been hiding in there since this is likely a rare event.

I'm going to set up a test environment to replicate this issue this afternoon after work.

Thanks again!

[Tbone542]

Excellent...

Thanks again,
Ted