airflow 2.6.2 - Dags disappears from UI and airflow API for dag triggers gives 404 error intermittently for dynamically generated dags

[kraj007]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.6.2

What happened?

Hi Team,

We have upgraded our airflow version from 1.10.10 to 2.6.2 recently.
We have one dag which reads from DB and launches multiple dags , its stores all these dynamic values in airflow variable.
Our DB backend is RDS mysql 8 version.
We are launching 50+ dags dynamically from this one dag and also we have 2 separate dags which have dedicated dag files.
We are intermittently get issues some time where on UI we only see 2 dags which are launched by dag files.
All other dags which are launched by dynamic dag are not visible, within few seconds they appear again while in background they are running fine.

We use Airflow APIs to trigger dags from some external code , but in code also we are getting intermittent failures with response 404. This happens for some seconds.

We tried to check the scheduler and worker logs but could not find any relevant info. We are trying to reproduce this scenarios but unable to do so because we are not sure what condition triggers this intermittent failures.

Please help us with any workaround or quick fix to handle this issue if this is some known issue reported for this version

What you think should happen instead?

We dont have particular logs but on UI dags disappears for some seconds and appears again.

How to reproduce

We are trying to reproduce this scenarios but unable to do so because we are not sure what condition triggers this intermittent failures.

Operating System

AWS ec2

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

We are using airflow dockerfile and deploy it via AWS ECS service.

Anything else?

This occurs intermittently now. This is observed on airflow 2.6.2 version,
We used to have same dynamic dag generation logic on 1.10.10 version but did not face any such issue previously

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

You are not following best practices - you should avoid running long running operations during DAG parsing (i.e. top-level code execution) - follow the documetnation to fix it - https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code - simply sometimes parsing of your DAG file takes too long.

You can modify various parameters in airflow configuration for example https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-file-processor-timeout - but this is a band-aid and you should optimize your DAG parsing time - to not reach out to DB every time DAG is parsed. Vaarious techniques that you can use are described in https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html

Converting to a discussion, as this is just not following good practices.

[kraj007]

Thanks @potiuk for suggestions !
I have few queries apart from above suggestions -

1. If there any spike in memory or cpu at DB level , does this impacts the dag parsing issue or will this cause this issue to happen?
2. In above suggestions , its mentioned to have cache so that DB calls will be minimized but cache feature is available in version 2.7 onwards, do we need to upgrade airflow ? will it solve the issue ?

[kraj007]

I saw following logs in scheduler-

DAG abc_every_hour is missing and will be deactivated.
ERROR�[0m - Process timed out, PID: 4996�
airflow.exceptions.AirflowTaskTimeout: Timeout, PID: 4996
INFO - Deactivated 71 DAGs which are no longer present in file.

On UI no dags were appearing. In airflow variable - I could see the values present

[potiuk]

1. If there any spike in memory or cpu at DB level , does this impacts the dag parsing issue or will this cause this issue to happen?

Spikes in resource useage ALWAYS impact things. If you share workload of different processes and pods on the same machine, they can impact each other. This is just common engineering knowledge, nothing specific to Airflow (so called "noisy neighbour" effect. Your job as deployment manager is to make your deployment in the way to minimise that - Airflow is agnostic to that - as explained on our installation page https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html where we explain Deployment Manager responsibilities - it's the Deployment Mnager's job to setup monitoring and deploy their resources and manage workflows. As for any other applications Deployment Manager should setup monitoring, that they usuallu use, observe resource usage, and if they see bottlenecks of resources they should properly plan their deployment to prevent them. There are no specific guidelines for that because the resource usage largely depend on what your code does (as you experienced first-hand) - so you need to adjust your resources to your case.

2. In above suggestions , its mentioned to have cache so that DB calls will be minimized but cache feature is available in version 2.7 onwards, do we need to upgrade airflow ? will it solve the issue ?

This cache is for using Airflow Variables. Upgrading is always good. But in your case you have to redesign how your Dynamic DAGs are generated following the best practices and examples explained in the docs I shared. You are not doing it now. You have to change the way your DAGs are constructed.

INFO - Deactivated 71 DAGs which are no longer present in file.
On UI no dags were appearing. In airflow variable - I could see the values present

This means that resulting of parsing your DAG files is not stable. Sometimes they produce the dynamic DAGs you expect and sometimes they don't. You need to debug it and make sure that every time you parse your DAG files, the result is the same set of DAGs. This is also why we do not recommend to pull some data and depend on external factors. It might be for example that the files or other data you are pulling when you parse the DAGs is changing, disappearing, is skipped on error or maybe the data is just partially available while you are pulling it.

Your job when writing DAG as DAG author is that EVERY TIME the DAG is parsed, it produces the same set of DAGs if you expect stable set of DAGs being available. Anything that causes instabilties and variable results should be tracked and eliminated. What it is - only you can tell by looking at what your DAG does and where the instabilities might come from.,