First check
Prefect Version
2.x
Describe the current behavior
Work-Queue Behavior
Each prefect work-pool can have any number of work-queues associated to it. Each work-queue can be assigned an integer priority by using the --priority flag in the prefect CLI:
prefect work-queue create my-queue --pool my-pool --priority 1
Worker Behavior
When starting a prefect worker, you can select one or more work-queues that the worker should pull work from by using the --work-queue flag in the prefect CLI:
prefect worker start --name my-worker --pool my-pool --work-queue my-queue
Describe the proposed behavior
Worker Behavior
When starting a prefect worker, you can select one or more work-queues that the worker should pull work from together with custom priorities. I am hoping for something along the lines of:
prefect worker start --name my-worker --pool my-pool --work-queue-priority-config "{'queue-1': 1, 'queue-2': 2}"
Example Use
This enhancement would allow different workers to poll a single work-pool but prioritize the work differently.
As an example, our current use case is this:
- We use prefect to orchestrate an ML pipeline which is triggered by a consumer facing application. Our pipeline starts whenever a user initiates a video upload to our site.
- Flow 1 will copy the user video onto one of our GPU servers for processing. We have many servers so we choose a random one. Each GPU server has it's own local hard drive which is NFS mounted onto all of the other GPU servers.
- After the upload is complete, we kickoff flow 2. Flow 2 will run our series of ML models on the video.
We would like to run one worker on each of our GPU servers and have all of those workers accept jobs from a common "gpu_jobs" work-pool. Within the gpu_jobs pool, we would like to divide the jobs into different work-queues depending on where the file is actually located (i.e. one work-queue for each server). We would like each worker to give the highest priority to it's own work-queue, but accept any other jobs with a lower priority.
For example, we would like to do:
prefect worker start --name server-1 --pool gpu_jobs --work-queue-priority-config "{'queue-1': 1, 'queue-2': 2}"prefect worker start --name server-2 --pool gpu_jobs --work-queue-priority-config "{'queue-1': 2, 'queue-2': 1}"
Additional context
No response
First check
Prefect Version
2.x
Describe the current behavior
Work-Queue Behavior
Each prefect work-pool can have any number of work-queues associated to it. Each work-queue can be assigned an integer priority by using the
--priorityflag in the prefect CLI:prefect work-queue create my-queue --pool my-pool --priority 1Worker Behavior
When starting a prefect worker, you can select one or more work-queues that the worker should pull work from by using the
--work-queueflag in the prefect CLI:prefect worker start --name my-worker --pool my-pool --work-queue my-queueDescribe the proposed behavior
Worker Behavior
When starting a prefect worker, you can select one or more work-queues that the worker should pull work from together with custom priorities. I am hoping for something along the lines of:
prefect worker start --name my-worker --pool my-pool --work-queue-priority-config "{'queue-1': 1, 'queue-2': 2}"Example Use
This enhancement would allow different workers to poll a single work-pool but prioritize the work differently.
As an example, our current use case is this:
We would like to run one worker on each of our GPU servers and have all of those workers accept jobs from a common "gpu_jobs" work-pool. Within the gpu_jobs pool, we would like to divide the jobs into different work-queues depending on where the file is actually located (i.e. one work-queue for each server). We would like each worker to give the highest priority to it's own work-queue, but accept any other jobs with a lower priority.
For example, we would like to do:
prefect worker start --name server-1 --pool gpu_jobs --work-queue-priority-config "{'queue-1': 1, 'queue-2': 2}"prefect worker start --name server-2 --pool gpu_jobs --work-queue-priority-config "{'queue-1': 2, 'queue-2': 1}"Additional context
No response