how to efficiently generate thousands of kubernetesPodOperator tasks

[talna-evogene]

Hi everyone
I have a DAG that contains a task that should run 8000 times with different parameters.
Currently this DAG gets the webserver stuck and sometimes the scheduler and random tasks fails.
I wanted to know if there is a better way to generate the tasks / some parameter tweaking might help.
would appreciate any help,

technical details:
We have airflow 2.0 running on kubernetes.
We have made these changes:
increased the scheduler to use 3 replicas
changes in the configuration:
AIRFLOW__CORE__SQL_ALCHEMY_POOL_SIZE 1000
AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG 1000
AIRFLOW__CORE__PARALLELISM 1000
AIRFLOW_CORE_SQL_ALCHEMY_MAX_OVERFLOW 60
AIRFLOW__CORE__DAG_CONCURRENCY 1000
we also increased pgbouncer connection limits accordingly.

I generated this task inside a task group using a loop:

resources=k8s.V1ResourceRequirements(
    requests={
        'memory': '1G',
        'cpu': '1'
    },
    limits={
        'memory': '2G',
        'cpu': 2,
    }
)
with TaskGroup('randoms') as randoms:
    for i in range(8000):
        k = KubernetesPodOperator(
            image="ubuntu:16.04",
            cmds=["/bin/sh", "-c", f"echo {i};sleep 20m"],
            arguments=[],
            name=f"airflow-test-pod-{i}",
            task_id=f"task_{i}",
            is_delete_operator_pod=True,
            dag=dag,
            resources=resources
        )

[potiuk]

Well. I am not sure if Airflow is limitation here. Are you sure you have 1 TB of memory in total and 1000 CPUs to run 1000 concurrent KubernetesPodOperators on it ? Have you checked if you have the right cluster to run that much of a workload?

[talna-evogene]

I use a pool that limits the number of tasks to 1000. If I change the loop to 1000 instead of 8000 it works fine,
that's why I though there may be an issue with rendering the DAG.

[potiuk]

Rendering the DAG might indeed be a problem if it is a single HUGE task Group. Maybe you can split it into several and see if that helps?

[potiuk]

I do recall a discussion on slack when someone complained that really big Task Groups cannot be rendered. This might be an issue with trying to fit all 8000 taks in memory while rendering and the rendering library.

[talna-evogene]

ok, I'll try to split it into a number of smaller taskgroups. I wonder if increasing resources to the webserver / scheduelers could help here.