-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Set the failed/completed jobs automatically removed after a period time #854
Comments
For what it's worth, we just run |
@mscharley thanks. But I'm wondering why we don't have a function to let us be able to set a permanent process for the queue to remove the jobs after a period time? Could we consider to have an improvement on this? |
Sure, and I 👍 'd this issue too. Just letting people know of workarounds in the meantime. |
We store our job details elsewhere so this is just duplicated data being stored. This is a much wanted feature from me as well :) |
@mscharley I wanted to clear jobs from failed state and I tried |
Without this feature the amount of data in this increases to infinity, unless one of the workarounds is implemented of course. It would be really great if it was possible to retain statistics (number of jobs processed), without having to keep the actual job details around. |
There is a As for retaining statistics for jobs without keeping the jobs around, I think that's relatively easy to do in userland by tracking the 'completed' and 'failed' events on the queue. Just keep increment some value in redis. |
The The problem with implementing the statistics in userland is that there is already statistics provided by bull, and that info would become incorrect when deleting (old) jobs. |
@axos88 it does not remove it immediately, please check the documentation: https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queueadd |
@manast you can keep around "X" number of items, but cannot set a time limit for them. But you're right in that what I am gunning here is actualy to extend that syntax to allow |
yes, however Date is also a sword with two edges, if you retain based on date you may end filling up all redis memory. |
@manast, Well yes, of course, but that's a decision that needs to be weighed by the user/programmer and the approariate action needs to be taken. And of course the two can be combined saying hold the data of a week, but up to maximum 100 jobs, or the other way around keep up to 100 jobs in redis, but delete them after a week, even if there are less than 100 to clean up unuseful data. |
For the reference, this seems to be implemented, if I understand this issue correctly. https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#keepjobs-options, bull/lib/commands/moveToFinished-9.lua Lines 119 to 132 in 4b8a386
{
removeOnComplete: { age: 86400 * 5 }, // remove completed jobs after 5 days
} |
Hi all,
Do we have an option in Bull allows to remove the failed/completed jobs in queue after a period time (1 day/week/month)? I think it's necessary because it can reduce the number items in Redis storage.
Thanks in advanced.
Tin
The text was updated successfully, but these errors were encountered: