Deferring a periodic job to multiple queues does not work as expected

[elemoine]

Let's say I want per worker server periodic tasks. Naturally I'd be tempted to use per-server queues:

@app.periodic(cron="*/5 * * * *")
@app.task(queue=host_settings.queue_name)
def my_task(timestamp):
    …

But, as I understand it, the procrastinate_defer_periodic_job function will refuse to defer a periodic job if a periodic job with the same name and same defer timestamp was already deferred, even if the periodic job is not deferred to the same queue as the one already deferred.

Shouldn't the procrastinate_periodic_defers table include a queue_name column? And shouldn't the unique constraint be UNIQUE (task_name, queue_name, defer_timestamp) rather than UNIQUE (task_name, defer_timestamp)?

[ewjoachim]

It's loosely linked to #260 I'd say.

As of now, the interaction between queue and task name is not always clear. In your example, I think what was imagined was a different task name AND queue. But you're right, we could improve this (and your solutions seems A-OK)

Generally, it looks like anywhere a "task" needs to be unambiguously identified, the combination of queue and task name should be used (whereas the task name alone should only be used to find the corresponding callable to launch)

[elemoine]

Great summary @ewjoachim!

[ewjoachim]

This could maybe be "Good for newcomers" too but I'm afraid there are too many unclear things.

[ewjoachim]

Anyone browsing this ticket might want to have a look at https://github.com/procrastinate-org/procrastinate/releases/tag/0.23.0