Debug cron that doesn't restart after Redis crash

[rileyai-dev]

I am trying to debug an issue I have with cron. The job is set to occur once every day and it works as expected. The problem is that when my Redis instance restarts, the cron job doesn't restart.

I've set up a Redis client to listen to when Redis is ready and it gets reconnected after restart. In addition, I can see that the cron job is still in Redis (from the bull-arena UI). I have tried to reproduce the bug locally but I wasn't able to.

My question is would it be possible to set up a listener on the Queue, QueueScheduler and Worker to see when they get active/ready after a Redis disconnection? That would help debug any connection issue.

[manast]

A cron job produces delayed jobs, so you can just check if your next repetition is in the delay status. If it is, as long as you have a QueueScheduler active connected to the Redis instance it should just pick the delayed job when it is time to run it and move it to the wait list.

[manast]

So the way to debug this is to check if there is such a delay job and if the scheduler is running.

[rileyai-dev]

@manast Thanks but how can I check that the scheduler has reconnected or if the worker is also connected?

[manast]

you can use a external redis connection with your QueueScheduler and then attach listeners to it to check the connetion events: https://github.com/luin/ioredis#connection-events

[rileyai-dev]

@manast Any doc on how I can use an external redis connection to a QueueScheduler?

[manast]

https://docs.bullmq.io/guide/connections

[rileyai-dev]

@manast Sorry to bother but using and external connection for QueueScheduler is not documented. Does it mean that it uses the same connection as Queue or does it work like Queue for initialization?

[manast]

Al classes have the same connection semantics you can check the api reference.

[rileyai-dev]

@manast I've tried to add an external connection but for some reason, the prefix gets overridden by Bull... if it possible to use a connection with a specific keyPrefix and that BullMQ adds its default "bull:" prefix? It gets very confusing since we expect the keyPrefix of the connection to be taken into account:

const connection = new Redis(
    {keyPrefix:'myPrefix:', port: address.port, host: address.hostname}
)

const queue = new Queue('myQueue', {connection});

Should I post another issue for this?

[manast]

BullMQ adds its own prefix, you can override it instead and ignore ioredis prefix functionality.

[rileyai-dev]

@manast Thank you so much for your responsiveness, it really helps integrating this library!