How to avoid CLUSTERDOWN and UNBLOCKED?

[thelinuxlich]

Revisiting this problem, I asked on the redis google group about it and here it is what @antirez said:

Hello,

CLUSTERDOWN is a transient error that happens when at least one master
node is down. If you want the partial portion of the cluster which is
still up to run regardless of a set of hash slots not covered, there
is an option inside the example "redis.conf" file, doing exactly this.

UNBLOCKED is unavoidable since it is delivered to clients that are
blocked into lists that are moved from a different master because of
resharding. We don't want them to wait forever for something that will
never happen, since those lists are not moved into a different master.

So CLUSTERDOWN should be handled by the application and or at client
level directly by retrying. UNBLOCKED should be handled rescanning the
config with CLUSTER SLOTS and connecting to the right node.

Cheers,
Salvatore

Configuration-wise, I've set my cluster with "cluster-require-full-coverage" to "no" so I just need to know how to cover this on the application side

[luin]

It seems that we can just resend the command when a CLUSTERDOWN error is received. Thank you for the information.

If you want to handle CLUSTERDOWN on the application side, you have to catch all errors returned from Redis and resend the command if the error is CLUSTERDOWN.

[thelinuxlich]

I'm also getting a lot of these:

ReplyError: EXECABORT Transaction discarded because of previous errors.

[luin]

You can get previous errors by error.previousErrors to see what they are. For instance:

redis.multi().set('foo').get('foo').exec().catch(function (err) {
  console.log(err.previousErrors);
});

[thelinuxlich]

And by the way, if the library has the autoResendUnfulfilledCommands option enabled by default, shouldn't it resend automatically after recovering from CLUSTERDOWN?

[luin]

autoResendUnfulfilledCommands is used to send unfulfilled commands after a reconnection. Since CLUSTERDOWN is caused by a master's being offline, we may use clusterRetryStrategy option to retry the node.

[thelinuxlich]

Now I have 3 masters and 9 slaves, let's keep this open for some days so I can see if those errors persist. I've also configured the cluster with "cluster-slave-validity-factor 0" and "cluster-migration-barrier 1"

[thelinuxlich]

good reference: http://redis.io/presentation/Redis_Cluster.pdf

[thelinuxlich]

even after increasing slaves and changing config to be more available, I'm receiving EXECABORT ocasionally with this previousErrors:

previousErrors:
 [ { [ReplyError: MOVED 1684 192.168.0.1:7000]
 name: 'ReplyError',
 message: 'MOVED 1684 192.168.0.1:7000',
command: [Object] },
{ [ReplyError: MOVED 1684 192.168.0.1:7000]
name: 'ReplyError',
 message: 'MOVED 1684 192.168.0.1:7000',
 command: [Object] } ] } 

[AVVS]

@thelinuxlich I believe you are trying to perform multi-key operations on the cluster, therefore you get the following errors

If you need to that, you need to make sure they have the same hash, ie {somehash}your-key-name1, {somehash}your-key-name2 etc

[thelinuxlich]

You mean, using multi(), right?

[thelinuxlich]

My code has one transaction:

redis.multi().setnx(new_key, possible_new_session_id).expire(new_key, 1800).exec()

[luin]

Is there a resharding or failover happens after cluster has been initialized?

[thelinuxlich]

no, probably the multi() is not going to the right node

[AVVS]

In that case multi() is fine (same key -> new_key), but the error says that the hash, that was resolved to new_key is now on another machine in the cluster. These errors should really be handled by the library. What the error says is that, hey, your hash slot caching is wrong, it needs to be updated and the operation needs to be retried.

@luin, please take a look at this, as I believe it needs to be improved. IE, MOVED reply must be handled.

On May 6, 2015, at 10:03 PM, Alisson Cavalcante Agiani notifications@github.com wrote:

My code has one transaction:

redis.multi().setnx(new_key, possible_new_session_id).expire(new_key, 1800).exec()
—
Reply to this email directly or view it on GitHub #28 (comment).

[thelinuxlich]

By the way, I didn't have this problem with a similar library: https://github.com/thunks/thunk-redis

Their API forces you to set the key explicitly on the multi() and exec() methods

[luin]

Yes, you are right. ioredis should be able to handle these MOVED errors. I'll try to fix these errors tonight.

[AVVS]

@thelinuxlich thunk-redis takes hash from the first key, and then applies it to all the operations in the multi, but they don't make use of pipeline. Here the first key from pipeline is taken and then the hash is applied to the whole pipeline.

Beside that its pretty much the same, except that ioredis seems cleaner (and with a bug not handling MOVED responses 💨)

[luin]

@thelinuxlich As stated in the README, ioredis will use the first key in the pipeline queue to calculate the slot. So the problem here isn't ioredis use the wrong key, instead is ioredis doesn't handle MOVED errors properly in the transaction.

[thelinuxlich]

ok, maybe refreshAfterFails = 1 can solve the issue?

[luin]

@thelinuxlich No, it doesn't help since it's a bug of ioredis. I'll fix it soon :-)

[thelinuxlich]

Dont know if it is luck, but since setting refreshAfterFails to 1 no error happened

[luin]

@thelinuxlich That's interesting. However I'm implementing a more stable transaction strategy in cluster mode.

[thelinuxlich]

let's test it ;)

[thelinuxlich]

still getting MOVED errors :(

[luin]

@thelinuxlich Yes, this commit just fixed CLUSTERDOWN errors, and I'm still working on handling MOVED errors :-)

[luin]

There is lot of work to do to implement a stable transaction in cluster mode. However the job is getting done :-)
Pull request #33 should handle MOVED, ASK and CLUSTERDOWN error properly. I'm writing more tests for it and will ship it in the next version. Welcome to do some tests if you have time.

[thelinuxlich]

Great, every new release I'm always testing :)

[luin]

Released in 1.3.0

[thelinuxlich]

No errors so far...

[thelinuxlich]

Seems fixed!