Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running cron every 5 minutes to NOT all run at the same time #2610

Closed
microSoftware opened this issue Jun 4, 2023 · 1 comment
Closed

Running cron every 5 minutes to NOT all run at the same time #2610

microSoftware opened this issue Jun 4, 2023 · 1 comment

Comments

@microSoftware
Copy link

Bull v4.10.4

I tried adding random delay from 0 to 5mn but doesn't work.
I tried adding new task at 0 to 20 seconds interval. But every time, the task say "will run at 3:05" or whatever multiple of 5: 3:10, 3:15, 3:20... So basically worker are idle for 5mn and then when it's 3:05 , all the workers execute at the same time and of course it takes more than 1mn to execute all the tasks so it's bad.

What i'd like is to evenly distribute the task in time. So the workers are always working.
So if i add a task at 3h08:10sec every 5mn. Then the task will be added for execution at 3h13:10sec and then 3h18:10
if i add another task at 3h09:30sec every 5mn. Then the task will be added for execution at 3h14:10sec and then 3h19:30

So naturally the task get evenly distributed overtime.

Which is not the case now. Now, if i add the task at 3h08 or 3h09. It will be executed at exactly 3h15 and then 3h20.
So all the task get executed at exactly the same time. So when you have thousands of tasks to be executed at the same time it's really not ideal.

Is there a way around this ?

Note: i use bull dashboard to check when the task will be executed in the future with the delayed tab.

@manast
Copy link
Member

manast commented Jun 4, 2023

You can use the offset option in BullMQ to affect the time bin when a job is executed: https://api.docs.bullmq.io/interfaces/RepeatOptions.html#offset
In Bull, this option does not exist and will not be added as it is in maintenance mode.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants