Airflow UI stops refreshing the DAG status

[pavelpi]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Airflow versions 2.6.1, 2.6.2
When we run a DAG in Airflow, after a few seconds frontend stops updating DAg tasks statuses on UI and we cannot see DAG processing progress. When we watch communication between frontend and backend, we can see that suddenly the server started to report 404 on grid_data?dag_id=<dag_id>&num_runs=25 with {"error": "can't find dag <dag_id>"}. It reports 404 twice and stops all further requests.
Refresh of the page causes page forward to DAGs landing page with error telling DAG <dag_id> seems to be missing from DagBag.
After a while (~1 minute) the DAG is discoverable again. In the background the DAG finishes normally.

What you think should happen instead

The web frontend should keep reporting the DAG progress.

How to reproduce

Run any long running DAG and watch its status.

Operating System

python:3.10-slim-bullseye

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.0.0
apache-airflow-providers-apache-spark==4.0.1
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-cncf-kubernetes==6.1.0
apache-airflow-providers-common-sql==1.4.0
apache-airflow-providers-elasticsearch==4.4.0
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-google==10.0.0
apache-airflow-providers-grpc==3.1.0
apache-airflow-providers-hashicorp==3.3.1
apache-airflow-providers-http==4.3.0
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-microsoft-azure==6.0.0
apache-airflow-providers-microsoft-mssql==3.3.2
apache-airflow-providers-mysql==5.0.0
apache-airflow-providers-odbc==3.2.1
apache-airflow-providers-opsgenie==5.0.0
apache-airflow-providers-oracle==3.6.0
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-redis==3.1.0
apache-airflow-providers-sftp==4.2.4
apache-airflow-providers-sqlite==3.3.2
apache-airflow-providers-ssh==3.6.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jscheffl]

This feels like a side effect that if DAG parsing has performance problems, DAGs might dis-appear from the system until re-parsed. In case of Error the UI stops refreshing.

Can you please check your DAG parser performance and if DAGs dis- and re-appear?

[pavelpi]

Hello @jens-scheffler-bosch
thanks for quick reply!
I scaled up the environment to have 3 DAG processors, each one having 4 CPUs and 2GB memory. 70% of the time all CPUs are almost fully allocated. There are periods when I can see the same issue. It's better than when I had only 2 processors, 1 CPU each. I also tried 1 processor running on 8CPU/8GB RAM machine, but this did not help much. Memory does not seem to be a problem - processors need about 400MB each regardless of number of processors or #CPUs.
Is there any recommended setup or environment setting that would help us?
We have 137 DAGs, DAG processor processes them between 0.5s and 2.0s each.
When we encounter the issue, although Airflow reports missing DAG, it is still available in the list of DAGs.

[jscheffl]

Most important question: Problem gone by increasing DagProcessor resources? Otherwise this should be fixed first.

Besides throwing more hardware on the problem you can also configure other timeouts when DAGs are purged if not re-parsed. See configuration reference for options.
Otherwise if you see performance problems and DAG parsing is slow, would be good to take alook to best practices to improve DAG parsing performance. Typical root cause is Top level code (see https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code) and Variables which are adding delay because of DB round trips.

If you have taken a look to DAG processor logs and the times are really in the range of 0.5-2s per DAG then really something is slow in your DAGs. Either is it not good code or your DAGs are very complex. Good is if DAGs are parsed in <0.1 seconds... it is just the parsing of the DAG not the execution.

If okay, in general I'd propose to close the issue.

[Taragolis]

It would be nice if you could provide some DAG samples which might reproduce an issue, otherwise it is looks like more troubleshooting rather than actual issue.