Extend elastic DAG with a binary tree, grid, star

[mik-laj]

Description

We have "elastic DAG" which we use in various performance tests. It allows you to generate any DAG using environment variables.
An example test that uses such a DAG looks like this

        with mock.patch.dict("os.environ", {
            "PERF_DAGS_COUNT": str(dag_count),
            "PERF_TASKS_COUNT": str(task_count),
            "PERF_START_AGO": start_ago,
            "PERF_SCHEDULE_INTERVAL": schedule_interval,
            "PERF_SHAPE": shape,
        }), conf_vars({
            ('scheduler', 'use_job_schedule'): 'True',
        }):
            dagbag = DagBag(dag_folder=ELASTIC_DAG_FILE, include_examples=False)
            with assert_queries_count(expected_query_count):
                processor = DagFileProcessor([], mock.MagicMock())
                processor._process_dags(dagbag.dags.values())

Source:
https://github.com/apache/airflow/blob/6bd2e59/tests/jobs/test_scheduler_job.py#L1175

This DAG File allows you to generate DAGs that contain all the tasks connected in a linear manner or have no connections.

It would be useful to add more complex DAGs, especially binary trees or/and grids or/and stars. This will allow us to test more complex cases more easily.

Use case / motivation

N/A

Related Issues

N/A

[alexandrecaze]

I am interested in taking this issue @mik-laj

[alexandrecaze]

@mik-laj What do you mean by grids and stars shaped DAGs ?

[mik-laj]

DAGs that look like the picture

[alexandrecaze]

Would we have a second layer for the star ?

or all children of task 1 ?

[mik-laj]

All children of task 1. This is needed to check if the tasks will be performed at the second iteration of the loop. Grid is needed to perform a stress test and at the same time not get lost in all dependencies. Binary tree will allow us to check cases more real to real situations.