Check concurrency limit before submitting tasks to TaskRunner #8539
Labels
concurrency
enhancement
An improvement of an existing feature
needs:design
Blocked by a need for an implementation outline
First check
Prefect Version
2.x
Describe the current behavior
Concurrency limits are checked after the task is submitted to the TasksRunner. For distributed task runners like Ray, this causes upscaling based on the total number of submitted tasks and not on the number of runnable tasks according to the concurrency limits. Especially for flows with a large number of tasks but rather low concurrency limits, this causes massive overprovisioning of compute resources.
Describe the proposed behavior
Only submit tasks to the task runner when free concurrency slots are available, which would lead to upscaling based on the concurrency limits and not the total number of tasks.
Example Use
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: