vdk-meta-jobs: implement a limit on starting jobs at once

[yonitoo]

What:
Create a check that checks for every job if there are more than a configurable amount of concurrent starting jobs (15 by default) in a Meta Jobs DAG. If so, the ones that exceed the limit would be delayed.

Why: We want to limit the number of simultaneously starting Meta jobs in order not to overload the SC

Tests: unit test is introduced

Signed-off-by: Yoan Salambashev ysalambashev@vmware.com

[antoniivanov]

Please link the github issue that you are addressing in this PR

[antoniivanov]

Good user experience is very important. The current solution moves the burden to the user to make sure they setup their dag. It is much better if we handle it for them.

If we automatically submit at max N jobs to start concurrently (regardless of how much has been setup).
How about other options :

Option A) Implementing a job type of queue logic : A job queue can help us manage the order in which jobs are executed. We can set a maximum number of jobs that can be added to the queue at any given time, and once this limit is reached, new jobs will be queued until there is the capacity to run them.
Option B) Using a semaphore: A semaphore is a synchronization tool that can be used to limit the number of concurrent executions of a critical section of code. You can use a semaphore to limit the number of jobs that can be started at any given time. Once the limit is reached, new jobs will be blocked until a running job completes.

The current solution, which involves raising an error and not allowing users to set up a DAG with a certain number of concurrent jobs, can be a good solution in cases where it is important to enforce strict limits on the system but we do not have such requirements.
Instead, this provides a more restrictive user experience. Users may be frustrated if they are unable to submit their jobs due to hard limits on the number of concurrent jobs, especially if they are not aware of these limits beforehand. This can lead to a poor user experience and may discourage users from using the plugin in the future.

This type of discussion is better done in the issue before the PR is open (though it's ok if it's here but it will slow down the PR review). I strongly recommend outlying your ideas in the github issue first.

[ivakoleva]

LGTM

[antoniivanov]

Btw the correct prefix in the commit mesage is just the name of the plugin vdk-meta-jobs: ...

We use the titles to auto-generate the release notes: https://github.com/vmware/versatile-data-kit/releases

[DeltaMichael]

It's a bit hard to understand what the code does. Just by looking at it and without having tried to run it, it should do the following:

1. Sort the DAG topologically
2. Get each level of the sorted DAG
3. If a level exceeds the max limit, raise an error

If this is correct, there should be a good library to use without having to implement it yourself. In case you still decide to implement it yourself, I suggest you break the code down into simple pieces, so it's more understandable, e.g. have one function that sorts, one that gets the levels and one that checks the length. This is a good approach regardless of how you handle the levels that are over the limit (raising an Error, @tozka's suggestion or something else)

[antoniivanov]

We met face-to-face (zoom to zoom) with @yonitoo and agreed on an approach where the MetaDag class ensures that there are at most N (configurable) concurrent jobs running by delaying the start of a new job if it exceeds the limit. We will reuse the already-built delay job functionality.

We also realized that the current validation approach would not work because it doesn't account for the fact that job duration could be different, and job start can be delayed for different reasons. As a result, it's possible that parts of the DAG may be ahead of other parts.

[yonitoo]

Thanks a lot for the feedback! @tozka @ivakoleva @DeltaMichael
After a discussion with Antoni today, we decided that from UX perspective, it would be best to just check before the start of each job if there are more than a configurable amount of jobs (15 by default) starting concurrently. If so, the rest of the jobs would be delayed. Therefore, the previous validation function is replaced by the aforementioned check as well as we would no longer throw an error. Also, the test is now remade and some long list initializations are replaced by list comprehension.

[yonitoo]

Actually, I just saw that @tozka already wrote the update, sorry for the spam.

[antoniivanov]

Improving your code reading abilities is essential for becoming a better programmer. So let me offer you some tips which I've found useful for me and I hope you find useful as well:

• Take your time: When you read code, don't rush through it. Take your time to understand each line and how it fits into the bigger picture.
• Identify the purpose of the code. Ask yourself what problem the code is trying to solve and how it fits into the larger project or system.
• Take some time to familiarize yourself with the data structures used in the code.
• For each method or class (or data structure) used: Read the method/class signature, Understand the parameters, understand the return value, Read the description/documentation, and when necessary read the implementation
• If you can do all that entirely in your mind, that's very good but it takes time to get used to it. You can also use a pen and paper or notebook editor - it may be helpful to take notes on what each line of code does.
• Use a debugger, just for reading purposes, to step through the code can help you understand how it works and what each line does.
• Practice, practice, practice - for example, reading code from other open-source projects is an excellent way to learn from other programmers.

[yonitoo]

Thanks for the tips, @tozka! Appreciate them!

[antoniivanov]

Looks good to me. Assuming the tests pass let's merge it.