feat(airflow): implement parallel-isolated mode

[IlyaFaer]

Towards: #931

Implements parallel-isolated decomposition mode.
Made in about the same way as the parallel mode, but the first source component is executed in parallel with others.

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	6b8f98c
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/65dce0981e89a000079d4c98

[IlyaFaer]

@rudolfix, I'm not 100% sure I understood what you mean by before execution pipeline will be forked under new name so state will be isolated. You mean that it'll be cloned with the drop() method here?

dlt/dlt/helpers/airflow_helper.py

Line 202 in cf2e886

task_pipeline = pipeline.drop()

It'll clone the pipeline, so seems to me it's what you mean by forking. Or maybe you mean that ForkPipe()?

[rudolfix]

@rudolfix, I'm not 100% sure I understood what you mean by before execution pipeline will be forked under new name so state will be isolated. You mean that it'll be cloned with the drop() method here?

dlt/dlt/helpers/airflow_helper.py

Line 202 in cf2e886

task_pipeline = pipeline.drop()

It'll clone the pipeline, so seems to me it's what you mean by forking. Or maybe you mean that ForkPipe()?

you are right! ticket was not clear. please take a look at updated requirements. also I added a requirement to add users if state is written in parallel mode

[rudolfix]

looks good! but spec was not clear so please see my comment...

[rudolfix]

I still have one issue (probably tickets were not clear). tests are really top!

[rudolfix]

@IlyaFaer I changed the parallel mode to not run in separate pipeline names. I think the ticket was not clear. otherwise good. test that attach to all partial pipelines is cool