[Feature]Prevent duplicate Task in the queue

[flotho]

Hi,

From time to time, jobs with the same Task signature are created by different methods.
This cause the queue to increase in an useless way.
Won't it be a good idea to prevent to create jobs if another one is existing in either 'pending' or 'started' state?
@guewen , what's your point of view on this one?

regards

[guewen]

@flotho The logic whether a job should be created or not depending if another job exists or not changes from job to job. At best we could add a way for devs to provide a function attached to the job, that would define this logic.

Keep in mind that jobs should be idempotent, and in some cases they can leave early if the work has been done already. I know this is not the same thing, but it works well enough usually in the use cases I had to implement. Anyway you have to take that in account because you could only merge "pending" "enqueued" or "started" jobs, so a identical job may already have been executed. On one side you lose some time executing a job that's already be done, on the other side, you slow down the creation of the job with extra checks. I prefer a fast delaying of jobs in the users transactions at the cost of a slower execution of those jobs in the background.

Do you have examples where you would really benefit from this feature? If yes, I'd welcome a pull request implementing a smart solution, which keep the current behavior by default, but allow a developer to define merge conditions for pending jobs.

[flotho]

Hi @guewen ,

Great thanks for this answer. Actually I understand your point of view that create fast and deliver fast will be the best.
As an example I managed 2 instance in odoo 8 and odoo9 with prestashop connector. (BTW I made some little proposal in the OCA repo that could be usefull).
The context is really simple, my customer succeed in its business and the number of order is increasing a lot. In the same way they develop their B2B activity and also have sale.order that have many, many lines... (more than a hundred).
So we have 2 channels, one coming from PS and the other form Odoo itself, and each validation triggered a job to update the stock on PS. If you work with 5 workers for the main channel, during high activities, products coming from the B2C channel could contain products coming from B2B channel so that in the queue you'll have 2 jobs and you could also have more than duplicates etc...
Additionally, if you consider that the products are created in PS with huge category tree structure, each product will have to check/create dependencies so you'll have a high volume of jobs.
All this stuff is slowing down all my process and from time to time the queue has to be reset manually.
In this case, I'm wondering if it won't be a good proposal to make the test in the run method so that each job won't run itself if he find another one having the same signature. In this case, it will skip itself and mostly set it to the done state.
What does it sounds like to you ?

[guewen]

Closed by #66