[Feature Request]: Limit concurrent jobs within a workflow

[dur3x]

Is there an existing feature request for this?

• I have searched the existing issues

What is your feature request?

Hi,

First thanks for all for this project, it's so nice 👍

I've been exploring the documentation and testing the system, and I noticed the following:

It is possible to set a max concurrent limit on jobs, events, or workflows globally.
However, I couldn’t find a way to limit the number of concurrent jobs inside a single workflow.

What I would like to do:

Start a workflow with many jobs defined.
Only have X jobs running concurrently, and as each finishes, the next one starts.

I also tried using some sort of controller, but it didn’t seem feasible (or maybe I missed something).

Would it be possible to add a feature to control concurrency at the workflow-level, or is there already a recommended approach for this?

Thanks!

Code of Conduct

• I agree to follow this project's Code of Conduct

[jhuckaby]

Thank you for your kind words!

So, just to clarify, does attaching a Max Concurrent Jobs Limiter Node inside the workflow solve your problem?

Combining this with a Max Queue Limit Node, as shown above, allows you to control concurrency for the event / job, and enable queuing.

Let me know if I am misunderstanding the ask.

[dur3x]

Thanks for your reply! I cannot test it before this Thursday but by reading it I’m not fully sure this solves my use case, so let me clarify: What I’m trying to achieve is limiting concurrency within a single workflow execution. For example: - A workflow launches 20 jobs (different or with different arguments) - I want only 5 jobs to run in parallel- When one finishes, another one starts (like an internal queue) My understanding is that the Max Concurrent Jobs Limiter Node and Max Queue Limit Node you mentioned apply more at the job/event level across executions, rather than controlling parallelism inside a single workflow instance. Thanks again!

[jhuckaby]

Ah, so that's a difference between an Event Node, and a Job Node in a Workflow.

• Event Nodes point to events which are globally defined, so the max concurrent limit applies to the event wherever it may be running.
• Job Nodes are unique to each workflow, so the max concurrent limit applies to the job node itself, inside the workflow where it is used.

So maybe if you use job nodes, it will.... oh wait, sorry, I just re-read your message:

My understanding is that the Max Concurrent Jobs Limiter Node and Max Queue Limit Node you mentioned apply more at the job/event level across executions, rather than controlling parallelism inside a single workflow instance.

Okay sorry, so both node types do apply their limits across all "executions" of the same workflow. So if you have 5 copies of the workflow running at once, the limits are unaware of that, and they instead just apply the limits based on how many copies of the "Event ID" or "Job Node ID" are running at once.

I may have to design something all new to support the thing you are describing.

[jhuckaby]

Okay, I've been thinking about this all day, and experimenting with different behaviors, and I've decided that your feature request here should be the way it always behaves, by design.

Workflow sub-jobs, when queued, will now act in their own "queue namespace" that includes the workflow parent job ID as part of the Queue ID.

Meaning, in effect, each workflow "node" will have it's own "queue", and they are not shared between execution runs, nor are they global. I think this will greatly reduce confusion overall.

So if you run 5 workflows in parallel, the individual queuing nodes inside each one will still behave correctly, and not be blocked by other jobs running outside of their namespace. They will each adhere to their own limit nodes, if present.

Obviously you can still set a limit on the number of workflows that can run in parallel. That hasn't changed. I've only changed the behavior of sub-nodes inside a workflow, and how they handle queuing.

I hope this makes sense. In short, you're getting exactly what you asked for in this feature request 😊

The code is complete and checked in, and will go out with the next release (trying to shoot for weekly releases now).