Error: Sending custom commands in pipeline is not supported in Cluster mode when use Cluster Mode #758

zetflo · 2017-10-25T10:03:42Z

When i use bull in redis cluster mode.

Configuration options

const opts = {
    createClient: function(type){
       return new redis.Cluster([
            ...
        ]);
    },
    prefix: '{}'
};

Create job it's worked but when i process job with queue.process,
I encounter the following error

Error: Sending custom commands in pipeline is not supported in Cluster mode.
    at Pipeline.exec (\node_modules\ioredis\lib\pipeline.js:249:19)
    at Pipeline.pipeline.exec (\node_modules\ioredis\lib\transaction.js:34:26)
    at \node_modules\bull\lib\job.js:210:18

Apparently the pipelines are not compatible with cluster mode of redis.

Can you confirm this problem?
How to use bull with a cluster redis without encountering this problem?

The text was updated successfully, but these errors were encountered:

manast · 2017-10-25T10:58:19Z

ok, seems like it is a limitation of ioredis at least. I did not now about it: redis/ioredis#536
This requires some rewrite in order to make it work in cluster mode unfortunately.

ajwootto · 2019-03-20T22:01:59Z

@manast we are starting to run into this. Do you know what features specifically require pipelines? Is there some subset of features we can still use without them?

btzsoft · 2019-04-22T16:31:26Z

@manast I'm running in the same issue when calling job.moveToFailed(err). it's possible to do something else here? it's strange for me, because I use job.moveToCompleted() without any problems. thx!

manast · 2019-04-22T17:29:30Z

@btzsoft that is because ioredis does not support multi commands using custom lua scripts, and only moveToFailed currently uses that. In order to solve this issue the method should be rewritten to not use multi, which is a bit to work to do...

btzsoft · 2019-04-22T18:04:54Z

@manast oh no, do you have some plans in future to change this? if no can you give pls some hints where to take a look, maybe I will make a fork and will changes this, bcs it's really a big blocker for us. thank you for your time.

manast · 2019-04-22T18:59:06Z

you can check the implementation of moveToFailed, it is quite inconvenient to not being able to use multi in this case. The other alternative is to convince the authors of ioredis to fix this issue:
redis/ioredis#536

btzsoft · 2019-04-23T09:21:32Z

@manast just to better understand, why moveToFailed use multi command? and why doesn't work as moveToFinished. thx!

manast · 2019-04-23T18:25:22Z

@btzsoft I think you could understand this better if you check the source code, basically it is because moveToFailed has much more functionality such as retries, updating attempts, backoffs, etc.

joebowbeer · 2020-06-02T16:42:14Z

@manast Does this mean bull is incompatible with a Redis cluster-mode-enabled cluster until ioredis is enhanced? Is there no workaround?

In the meantime, can you update the Patterns doc, which provides a (broken?) example of how to use Redis Cluster?

https://github.com/OptimalBits/bull/blob/develop/PATTERNS.md#redis-cluster

akhil-karat · 2020-07-11T04:16:45Z

@manast - I am facing same problem. Works with local redis instance. But in our int and prod we use Redis cluster. When a job is added to queue process fails with error 'Sending custom commands in pipeline is not supported in Cluster mode'. Then job stalls. Is there a work around for this issue? It is really urgent for us. Kindly help.

joebowbeer · 2020-08-13T20:56:32Z

@manast Can you confirm that this is not an issue in BullMQ?

manast · 2020-08-14T08:35:57Z

@joebowbeer it is the same problem unfortunately. I was expecting ioredis to be fixed but for some reason they did not prioritize it. The alternative is to convert "moveToFailed" method in bull/bullMQ to one lua script, as you can see it is quite a bit of job, but it seems like it is the only way to solve this issue once and for all... will try to prioritize it.

manast · 2020-08-14T08:37:43Z

@akhil-karat

@manast - I am facing same problem. Works with local redis instance. But in our int and prod we use Redis cluster. When a job is added to queue process fails with error 'Sending custom commands in pipeline is not supported in Cluster mode'. Then job stalls. Is there a work around for this issue? It is really urgent for us. Kindly help.

Are you sure this happens when you add the job and not when the job fails?

joebowbeer · 2020-08-14T09:17:41Z

@manast Thanks for prioritizing this! I repo'd the issue in BullMQ as well.

By using just a single ioredis custom command invocation now, rather than a transaction including a custom command, we should have OptimalBits#758 resolved. Slightly simplified logic (hopefully).

Affects the following scripts: - moveToDelayed - moveToFinished - retryJob Allows for the `attemptsMade`, `stacktrace` and `message` fields to be updated through Lua scripts, rather than requiring a separate operation (or multiple operations) to set these fields. With the possibility of setting these fields within the context of a single script, we avoid the need to craft a transaction containing both HMSET and EVAL(SHA) commands to achieve atomicity. By shedding the need for a transaction, we are able to avoid redis/ioredis#536 and thus solve a common problem seen when targetting Redis clusters. This isn’t necessarily an ideal means of solving this problem, given we’re violating DRY principles by copying the same logic across scripts. It’d be ideal if the underlying ioredis restriction was removed (in a manner which doesn’t entail the performance cost of resubmitting scripts each time).

By using just a single ioredis custom command invocation now, rather than a transaction including a custom command, we should have OptimalBits#758 resolved. Slightly simplified logic (hopefully).

Affects the following scripts: - moveToDelayed - moveToFinished - retryJob Allows for the `attemptsMade`, `stacktrace` and `message` fields to be updated through Lua scripts, rather than requiring a separate operation (or multiple operations) to set these fields. With the possibility of setting these fields within the context of a single script, we avoid the need to craft a transaction containing both HMSET and EVAL(SHA) commands to achieve atomicity. By shedding the need for a transaction, we are able to avoid redis/ioredis#536 and thus solve a common problem seen when targetting Redis clusters. This isn’t necessarily an ideal means of solving this problem, given we’re violating DRY principles by copying the same logic across scripts. It’d be ideal if the underlying ioredis restriction was removed (in a manner which doesn’t entail the performance cost of resubmitting scripts each time).

By using just a single ioredis custom command invocation now, rather than a transaction including a custom command, we should have OptimalBits#758 resolved. Slightly simplified logic (hopefully).

Rua-Yuki · 2020-09-09T22:26:21Z

Super sorry for the excessive mentions all! (Was doing a bit of cleanup and of course GH has appended to this discussion for every instance of these commits.)

That said, if anyone urgently needs this working, a patch can be found in the fpm-git/bull/bug/fix-#758-redis-cluster-breakage branch.

Insert the usual "it works for us, but no warranties, etc." spiel here.

joebowbeer · 2020-10-23T15:50:19Z

@manast I think the ioredis issue is fixed in v4.19.0

huxwfun · 2020-12-11T07:39:32Z

@Rua-Yuki
it appears addBulk uses pipeline as well

Job.createBulk = function(queue, jobs) {
  jobs = jobs.map(job => new Job(queue, job.name, job.data, job.opts));

  return queue
    .isReady()
    .then(() => {
      const multi = queue.client.multi();

manast · 2020-12-11T07:55:54Z

Unfortunately the fix the did on ioredis breaks Bulls unit tests for repeatable jobs, so we cannot use that fix until we discover the reason for the breakage.

Rua-Yuki · 2020-12-12T06:43:18Z

@Rua-Yuki
it appears addBulk uses pipeline as well

Job.createBulk = function(queue, jobs) {
  jobs = jobs.map(job => new Job(queue, job.name, job.data, job.opts));

  return queue
    .isReady()
    .then(() => {
      const multi = queue.client.multi();

@huxwfun Many thanks for bringing this to my attention! Our use case only demanded a quick fix for the issue raised here with moveToFailed, though I can definitely see about adding support for pipeline-free addBulk as well (potentially sometime early next week).

Alternatively, #1933 can be used to get the benefits of ioredis 4.19.0+, which I believe should fix this sort of issue without requiring elimination of scripts from pipelines.

Unfortunately the fix the did on ioredis breaks Bulls unit tests for repeatable jobs, so we cannot use that fix until we discover the reason for the breakage.

I've just added #1933, which seems to have the repeatable job unit tests all happy and green with ioredis 4.19.0+.

Please don't hesitate to let me know if there are any further steps I should take, or anything I might do to help out. Thanks!

gurbaj5124871 · 2021-03-26T07:39:59Z

Is this issue resolved by updating the bull and ioredis because it seemed to fix it for me, should this issue be closed and resolved?

manast · 2021-03-26T07:53:20Z

@gurbaj5124871 I am not 100% sure. Maybe somebody else from this thread could verify it before we close.

manast added the enhancement label Nov 5, 2017

manast added the PRIO 1 label Aug 14, 2020

joebowbeer mentioned this issue Aug 14, 2020

Job.moveToFailed fails in Cluster mode taskforcesh/bullmq#257

Closed

Rua-Yuki mentioned this issue Dec 12, 2020

fix: resolve repeatable test suite breakage with ioredis 4.19.0+ #1933

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: Sending custom commands in pipeline is not supported in Cluster mode when use Cluster Mode #758

Error: Sending custom commands in pipeline is not supported in Cluster mode when use Cluster Mode #758

zetflo commented Oct 25, 2017

manast commented Oct 25, 2017

ajwootto commented Mar 20, 2019

btzsoft commented Apr 22, 2019

manast commented Apr 22, 2019

btzsoft commented Apr 22, 2019

manast commented Apr 22, 2019

btzsoft commented Apr 23, 2019

manast commented Apr 23, 2019

joebowbeer commented Jun 2, 2020 •

edited

akhil-karat commented Jul 11, 2020

joebowbeer commented Aug 13, 2020

manast commented Aug 14, 2020

manast commented Aug 14, 2020

joebowbeer commented Aug 14, 2020

Rua-Yuki commented Sep 9, 2020

joebowbeer commented Oct 23, 2020 •

edited

huxwfun commented Dec 11, 2020 •

edited

manast commented Dec 11, 2020

Rua-Yuki commented Dec 12, 2020

gurbaj5124871 commented Mar 26, 2021

manast commented Mar 26, 2021

Error: Sending custom commands in pipeline is not supported in Cluster mode when use Cluster Mode #758

Error: Sending custom commands in pipeline is not supported in Cluster mode when use Cluster Mode #758

Comments

zetflo commented Oct 25, 2017

manast commented Oct 25, 2017

ajwootto commented Mar 20, 2019

btzsoft commented Apr 22, 2019

manast commented Apr 22, 2019

btzsoft commented Apr 22, 2019

manast commented Apr 22, 2019

btzsoft commented Apr 23, 2019

manast commented Apr 23, 2019

joebowbeer commented Jun 2, 2020 • edited

akhil-karat commented Jul 11, 2020

joebowbeer commented Aug 13, 2020

manast commented Aug 14, 2020

manast commented Aug 14, 2020

joebowbeer commented Aug 14, 2020

Rua-Yuki commented Sep 9, 2020

joebowbeer commented Oct 23, 2020 • edited

huxwfun commented Dec 11, 2020 • edited

manast commented Dec 11, 2020

Rua-Yuki commented Dec 12, 2020

gurbaj5124871 commented Mar 26, 2021

manast commented Mar 26, 2021

joebowbeer commented Jun 2, 2020 •

edited

joebowbeer commented Oct 23, 2020 •

edited

huxwfun commented Dec 11, 2020 •

edited