A Hiden Key

[stillerrr]

In our redis cluster, we find a node using more memory than the others, we analyzed the rdb file of this node, and find this node has a large key. When we try to get the key, the node displays that it do not exist, the slot of the key is not belong to the node.
How can I deleted this key from the node and what caused this situation?
Thanks

[trevor211]

I think it may be caused by disruption of a migration.
Say we have the following 4 nodes: M1, R1, M2, R2.
M1 and M2 are masters, and R1 and R2 are replicas, and R1 replicates M1, R2 replicates M2.
That is
M1 --> R1
M2 --> R2

We migrate some slots from M1 to M2.
Say we are migrating some specific slot(e.g. slot 16380) from M1 to M2, and that slot hold 3 keys: key_a, key_b and key_c.
And assume that we already migrated key_a to M2. What if a client ask for the value of key_a?
Certainly it would got a ASK which gives a hint that key_a is currently at M2 and the client would ask for M2 for the key.
The client would first send a ASKING command to M2 and get the key key_a, and M2 would respond with the value of key_a.

What if M1 suddenly disrupts?
Well, since only masters save the migrating status. When R1 got promoted, it would never know that slot 16380(as assumpted before) is in a migrating status.
And when some client ask R1 for the value of key_a, R1 would simply check from its local db instead of giving a ASK hint.
When some other client ask M2 for the value of key_a, it would got a hint that key_a is current in R1. But as a matter of fast, key_a is saved in M2's db, it just can't be accessed from client by normal command like GET. If you scan the db of M2, key_a would be spotted out.