Bug: Incompatible Workers Versions

[lukepolo]

We've started to get these errors, bullmq pro 1.2.2

ReplyError: ERR Error running script (call to f_54976b3a01720d05c4e64db764644c49b2979cc6): @user_script:82: WRONGTYPE Operation against a key holding the wrong kind of value
    at parseError (/opt/agent/main.js:2:3856421)
    at parseType (/opt/agent/main.js:2:3856430) {
  command: {
    name: 'evalsha',
    args: [
      '54976b3a01720d05c4e64db764644c49b2979cc6',
      '8',
      'agent-cloud:Snapshots:wait',
      'agent-cloud:Snapshots:active',
      'agent-cloud:Snapshots:priority',
      'agent-cloud:Snapshots:events',
      <Buffer@0x61b9cf0 de 00 04 a5 74 6f 6b 65 6e d9 24 36 32 34 34 33 66 63 39 2d 62 38 37 66 2d 34 65 30 34 2d 39 62 31 61 2d 65 38 34 39 36 39 35 61 31 64 36 32 ac 6c 6f ... 29 more bytes>
    ]
  }
}
      'agent-cloud:Snapshots:stalled',
      'agent-cloud:Snapshots:limiter',
      'agent-cloud:Snapshots:delayed',
      'agent-cloud:Snapshots:delay',
      'agent-cloud:Snapshots:',
      '1650580270008',
      '649273',
      <Buffer@0x61b9cf0 de 00 04 a5 74 6f 6b 65 6e d9 24 36 32 34 34 33 66 63 39 2d 62 38 37 66 2d 34 65 30 34 2d 39 62 31 61 2d 65 38 34 39 36 39 35 61 31 64 36 32 ac 6c 6f ... 29 more bytes>
    ]
  }
}

In the middle of debugging, were not 100% sure what the issue is as we have not changed implementations but restarted the worker for a release.

Hoping you've seen the error and its an easy fix (we'd rather not flushdb if we dont have to)

UPDATE:
we had to flushdb all to fix , after doing that (no code changes / restarting or anything) things started to work, we were in a non testing environment so had to do the hammer 🔨

[manast]

Could you please double-check this was the correct version? because version 2.0.0 includes a small breaking change that could lead to an error like this.

[lukepolo]

Current versions

"@taskforcesh/bullmq-pro@^1.2.2":
  version "1.4.1"
 resolved "https://npm.taskforce.sh/@taskforcesh/bullmq-pro/-/@taskforcesh/bullmq-pro-1.4.1.tgz#40d33b495af16c23222988e72629b8be260fc8d5"
  integrity sha512-KVm4XSRSKWemMHipS1My7DCudDs6zjuz0ZSrakHW/lf6xJqVz6HkBQHPpLrZ7ySH0VeI8fN0mbHJDnCbGzRjTg==
  dependencies:
    bullmq "1.80.4"
    lodash "^4.17.21"
    msgpackr "^1.5.1"
    rxjs "^6.6.7"
    tslib "^2.3.1"
    uuid "^8.3.2"
    ```
    
    
    ```
    bullmq@1.80.4:
  version "1.80.4"
  resolved "--retracted--
  integrity sha512-j3PyjU16gqmb3Md9QjMInJdbMvxIlSJx7mojtoP06LV9MfhzW75DkDrpSuJlF0H+0+u6MViV4hhaGTxky5OJWw==
  dependencies:
    cron-parser "^4.2.1"
    get-port "^5.1.1"
    glob "^7.2.0"
    ioredis "^4.28.5"
    lodash "^4.17.21"
    msgpackr "^1.4.6"
    semver "^6.3.0"
    tslib "^1.14.1"
    uuid "^8.3.2"
    ```

[lukepolo]

Also just to note, this release did not change bullmq-pro / bullmq or ioredis packages

ioredis@^4.28.5:
  version "4.28.5"
  resolved ""
  integrity sha512-3GYo0GJtLqgNXj4YhrisLaNNvWSNwSS2wS4OELGfGxH8I69+XfNdnmV1AyN+ZqMh0i7eX+SWjrwFKDBDgfBC1A==
  dependencies:
    cluster-key-slot "^1.1.0"
    debug "^4.3.1"
    denque "^1.1.0"
    lodash.defaults "^4.2.0"
    lodash.flatten "^4.4.0"
    lodash.isarguments "^3.1.0"
    p-map "^2.1.0"
    redis-commands "1.7.0"
    redis-errors "^1.2.0"
    redis-parser "^3.0.0"
    standard-as-callback "^2.1.0"
    ```

[roggervalf]

I'm checking the changelog and the ones that contains bullmq 1.80.4 are bullmq-pro 1.4.1 and 2.0.0

[lukepolo]

Hit the bug now locally, i think it may be due to something with stopping and starting workers....

Error extending lock  ReplyError: ERR Error running script (call to f_92b63b411a63b0e5484a78ef9b234b462411c815): @user_script:16: WRONGTYPE Operation against a key holding the wrong kind of value
    at Function.extendLock (/opt/qumulex/app/node_modules/bullmq/dist/cjs/classes/scripts.js:112:23)
    at async /opt/qumulex/app/node_modules/bullmq/dist/cjs/classes/worker.js:297:36 {
  command: {
    name: 'eval',
    args: [
      '--[[\n' +
        '  Extend lock and removes the job from the stalled set.\n' +
        '  Input:\n' +
        "    KEYS[1] 'lock',\n" +
        "    KEYS[2] 'stalled'\n" +
        '    ARGV[1]  token\n' +
        '    ARGV[2]  lock duration in milliseconds\n' +
        '    ARGV[3]  jobid\n' +
        '  Output:\n' +
        '    "1" if lock extented succesfully.\n' +
        ']]\n' +
        'local rcall = redis.call\n' +
        'if rcall("GET", KEYS[1]) == ARGV[1] then\n' +
        '  --   if rcall("SET", KEYS[1], ARGV[1], "PX", ARGV[2], "XX") then\n' +
        '  if rcall("SET", KEYS[1], ARGV[1], "PX", ARGV[2]) then\n' +
        '    rcall("ZREM", KEYS[2], ARGV[3])\n' +
        '    return 1\n' +
        '  end\n' +
        'end\n' +
        'return 0\n',
      '2',
      'clips:clip-convert:2:lock',
      'clips:clip-convert:stalled',
      '43a2762a-8bc6-41ba-8adf-74ccc0b09d1d',
      '30000',
      '2'
    ]
  }
}

Here is how we shutdown our workers :

process.once(DEV ? "SIGUSR2" : "SIGTERM", async () => {
        logger.warn(`Gracefully Shutting Down...`);
        logger.info(`---------------------------`);
        logger.warn(`Stopping Workers...`);
        await Promise.all(
          workers.map(async ({ queue, worker }) => {
            logger.info(`    [${queue}] stopping...`);
            await worker.stop().catch((error) => {
              logger.error(
                `        [${queue}] unable to shutdown worker gracefully`,
                error
              );
            });
            logger.info(`    [${queue}] stopped`);
          })
        );
        logger.info(`Stopped Workers`);
        logger.info(`---------------------------`);

        process.exit(0);
      });
      
      ```
      
      
      ```
  public queue: BullMQQueuer;
  protected scheduler: QueueScheduler;
....


      public async stop() {
    await this.scheduler.close();
    await this.queue.close();
  }

[manast]

@lukepolo we have analyzed this issue, and we can see that this is provoked by either mixing bullmq workers with bullmq-pro or mixing bullmq-pro workers from different versions. The teaching for us here is to implement some kind of protection mechanism so that you will get an error if trying to mix different versions of the libraries with the same queue.

[lukepolo]

Thank you for finding that out!!!! Thats a huge help , I’ll make sure we are using the correct versions ! Will let you know what I find

[lukepolo]

we found that it was updated via a yarn.lock, so we decided to hard lock the version .

Question is , how do we upgrade if our old jobs will fail?

[roggervalf]

hi @lukepolo, sorry for the delay, we are working on migration changes to work for old jobs as well, we will let you know the version to try it

[lukepolo]

No worries thanks for tackling this you guys are awesome !

[lukepolo]

is this fixed in the most recent version ?

[roggervalf]

hi @lukepolo, the fix is since version 2.0.2, also you can check all the releases on https://docs.bullmq.io/bullmq-pro/changelog