Airflow job is not showing up on the UI but visible in Task details

[jigs031]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

We have spined up the Airflow from Helm chart 1.6.0. We have defined AirflowExcecutor as "KubernetesExecutor" in our config map.

System works fine when number of parallel runs are less than, but when parallel runs increase - few of the PODS randomly runs on "LocalExecutor" and these task are not visible on Airflow DAG Run UI screen. After the long debugging we identified that it was dues to the environment variable "AIRFLOW__CORE__EXECUTOR " hard-coded to "LocalExecutor" in chart/files/pod-template-file.kubernetes-helm-yaml.

We tried to hard-coded as "KubernetesExecutor", then all the Jobs are working as expected.

What you think should happen instead

Like other configuration, Variable AIRFLOW__CORE__EXECUTOR in the pod template file should not be hardcoded (there is a possibility to add it via .Values.<variable_name>).

How to reproduce

Install the helm chart and define - "AIRFLOW__CORE__EXECUTOR == "KubernetesExecutor" in ConfigMap.

Run single job multiple times (atleast 20+ runs) within 10 seconds. You may observe that - not all job instances are visible on the screen. If we go on Task level details - you may find task are available but the Job are not register on Airflow UI/Database.

Operating System

Linux

Versions of Apache Airflow Providers

2.3.1

Deployment

Other 3rd-party Helm chart

Deployment details

Official Helm Chart version: 1.6.0

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

As already explained seeral times - AIRFLOW__CORE__EXECUTOR is always set to LocalExecutor in pod template file in case KubernetesExecutor is used. This is not a problem

You might experience other issue, but in order to be able to act on it, some logs, indication of what is going on and reproducible case need to be provided. Screenshots and logs from the tasks showing the problem you see should be added in order to see what really the problem is.

[potiuk]

Also. What I strongly suggest - before attempting to provide a reproducible case - upgrade Airflow. 2.3.1 has been followed by 2.3.2, 2.3.3, 2.3.4 - those are only bugfix releases to 2.3* line, and then there are many releases for 2.4, 2.5, 2.6 - all of them with several hundreds of bug fixes.

While you are completely "fixed" with your idea of "Executor" being hardcoded (which I already explained is not an issue), the reason for your problem might be different, but you have difficulty in explaining and diagnosing it. And it's more than likely it's fixed in some of those bugfix releases, or maybe in those minor releases afterwards.

So - before attempting to find a reproducible case - I strongly encourage you to upgrade Airflow - to at least latest version in 2.3.4 before doing so. this might save you (and anyone who would try to diagnose your issue) a lot of time.