Execute Flows Within a Work Queue Based on Priorities

[marvin-robot]

Opened from the Prefect Public Slack Community

giorgio.basile4: Hi, I have a question regarding queue priorities. I have a requirement that needs flow runs with "high-priority" to jump all other items in the deployment queue - basically switching from FIFO to LIFO.
I saw the https://docs.prefect.io/concepts/work-queues/#priority-queues|documentation on the topic but I don't think I would be able to support such use case. My understanding is that I would need two "low-priority" and "high-priority" queues, but if there is any item in the high-priority, no "low-priority" flow run should start, and I don't think it can be done easily.
Are priorities within the same queue something that would be supported in the future, or other ways to work it around? Thanks!

khuyen: This is an interesting feature request that could be useful for other users.

I’ll open an issue for this.

<@ULVA73B9P> open “Execute Flows Within a Work Queue Based on Priorities”

Original thread can be found here.

[stratosgear]

Even better would be some additional API calls that will allow us (after querying a specific work queue for the pending items) to promote or demote then.

Maybe by passing a command like make_first or make_last, or promote (which would make the 3rd item in the queue become 2nd, for example), or demote (well, the 3rd becoming 4th, if not last already).

Our use case is for some long lasting flows, that result in the queue accumulating a considerable amount of pending flows, that the end users would like to rearrange, while they wait for the already executing ones to finish.

That would be extra shinny!

Does this belong here, or should it be a separate issue?

[github-actions]

This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

[rbarrette]

+1, this is something that would be useful to my team as well, can this be reopened?

[giorgiobasile]

+1, this is something that would be useful to my team as well, can this be reopened?

@rbarrette just seen your comment, priorities are now handled natively in Prefect, checkout the Work Pools & Workers docs.