[Feature][framework] Throttling total number of pipelines running in parallel

[klesh]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently, the pipelines are launched right after their creation, this leads to the following concerns:

1. Users might want to limit the number of running pipelines to avoid race conditions.
2. Users might want to limit the resource that DevLake is consuming.
3. Running too many pipelines simultaneously might be too heavy for the IO system, which increases the probability of timeout exception.

@CamilleTeruel @hezyin Sending out requests in a short burst increases the probability of timeout exception.

Use case

1. Some users wanted to limit the number of pipelines running in parallel to exactly 1, so they could dump tables from our MySQL database to Starrocks without worrying about race-condition
2. Users might create pipelines as many as they need without crashing the system
3. Users could adjust the number to balance the speed and stability

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[likyh]

OK, I'll take it.