Enhance Concurrency and Cancellation Handling in Job Execution

[falvarez1]

Description:

This Pull Request introduces significant enhancements to how concurrency and cancellation are managed within job executions. Key changes include:

• Asynchronous Execution Enhancements: All job runs are now fully asynchronous, ensuring that operations are non-blocking and responsive throughout their execution paths.
• Improved Cancellation Support: Enhancements to cancellation token usage allow for more responsive cancellation of ongoing jobs. Cancellation checks are integrated more deeply into the job execution logic, allowing for quicker termination of jobs in response to cancellation requests.
• Concurrency Control with Priority Queue: Implemented a priority queue-based concurrency control mechanism. This allows multiple job instances to run in parallel while respecting priority constraints.

These changes aim to make the job execution process more efficient, responsive, and reliable, particularly under high load and in complex scheduling scenarios.

Future work, the concurrency control responsibilities still need to be moved into the JobExecutor , this shift will centralize how parallel executions are managed, simplifying maintenance and extending capabilities.

[falvarez1]

@linkdotnet how do you want contributions to be made? I see code being pushed directly to main, do you have a preference?

[linkdotnet]

@falvarez1 Thanks for providing that quality PR!

As I did mostly work alone I directly merged to main. The general rule of thumb is that if it isn't a breaking change it goes to main otherwise to v2, v3 and so on.

I do see some overlap with #21 - is this PR still "valid"? Or does it make sense to review and merge #21 first and see which bits are still missing?

[falvarez1]

Since #21 piggybacks on this one I suppose this one is unnecessary. I probably should have worked on the same branch but wasn't sure about your contributions preferences.

[falvarez1]

One thing to note, the Tests are broken at the moment. I didn't spend any time fixing them because I just wanted to have you review the design changes first.

[linkdotnet]

That is fine - they can be fixed afterward.

[falvarez1]

Closing this PR as it is superceded by #21.