Bug: exception when job processing returns immediately

[andrisi]

For some type of jobs I'd like to skip them if they've been added to the queue more then a minute ago. Just return from the job processing function, but if I do so, it throws an exception while in the waitUntilFinished() function. If I wait 200ms, there is no exception.

Error: Missing key for job bull:s-10-qd:6d7f769f-37cf-40a4-88ba-bf7aeb6ed971. isFinished
        at onFailed (C:\Develop\mastercompare\node_modules\bullmq\dist\classes\job.js:474:24)        
        at C:\Develop\mastercompare\node_modules\bullmq\dist\classes\job.js:496:21
        at processTicksAndRejections (node:internal/process/task_queues:96:5)

[manast]

Can you provide a complete use case that demonstrates the issue?

[andrisi]

I push jobs into a queue based on users' requests that need to be served quickly. If the workers fail and are restarted, lots of jobs like this flood them. I want the jobs to expire after a certain time so this does not happen. Users (or the UI on their behalf) will re-request the actions. The actual problem is that if a worker is too quick to return (immediately) than something seems to go wrong.

[manast]

@andrisi can you please provide a code example that reproduces the error?

[rysi3k]

Hi @manast
I prepared reproducing code for this error, as it occurs for my app as well.

const bullmq = require("bullmq");
const ioredis = require("ioredis");
const connection = new ioredis("redis://127.0.0.1", {enableReadyCheck: false, maxRetriesPerRequest: null});
// Reuse the ioredis instance
const myQueue = new bullmq.Queue('myqueue', { connection });
const myQueueEvents = new bullmq.QueueEvents('myqueue', { connection });

new bullmq.Worker('myqueue', async (job) => {
    return 'ok';
}, { connection });

myQueue.add('myqueue', {color: 'pink'}, {removeOnComplete: true, removeOnFail: 1000},).then(async (job) => {
    // simulate some CPU load/task:
    await new Promise((r) => setTimeout(r, 500));
   //
    console.log(await job.waitUntilFinished(myQueueEvents));
});

Thanks for help :)

Regards

[manast]

@rysi3k thanks for the code, I can reproduce it now.

[manast]

Ok, so the problem occurs because removeOnComplete is set to true. When job.waitUntilFinished is called, the job does not exist anymore in the Queue, and so the error is correctly reported. As I see it this is working as designed, the only solution is to increase the value of the removeOnComplete setting.

[manast]

I will close this issue now, also I encourage you to avoid using job.waitUntilFinished at all costs, here a post I wrote a while ago: https://blog.taskforce.sh/do-not-wait-for-your-jobs-to-complete/