504 Gateway Timeout when loading the grid view when there are too many past DAG runs

[zachliu]

Apache Airflow version

3.0.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When a DAG has too many past runs (e.g., an hourly DAG that has been running continuously for a year), the UI fails to load the grid view and returns a 504 Gateway Timeout, even though the request specifies limit=10.

I don't have an exact number to define "many past runs". But my Airflow 3 failed to load the grid view for a DAG that has over 8,000 DAG runs. After deleting ~5,000 DAG runs from the database (sadly searching & deleting DAG runs is removed from the UI in Airflow 3), the UI is able load the grid view but with a significant delay of 49.21s

What you think should happen instead?

With the proper limit, grid view should load pretty fast without 504 Gateway Timeout.

How to reproduce

Create a "minutely" dag to accumulate a few thousands of DAG runs and try to load the grid view

Operating System

LM 21.3 with docker python:3.12-slim-bookworm

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[tirkarthi]

It will be helpful to have below details to simulate this since this looks like a performance issue

1. Can you please share the endpoint that is timing out from API logs?
2. Are you seeing any slower query logs in the database end?
3. Can you please share a sample dag with details like tasks per dag, failure rate of the task instances

cc: @pierrejeambrun @bbovenzi

[pierrejeambrun]

i already did a bunch of optimization on the grid endpoint, but I can easily imagine why this endpoint remains slow for big dag run history, it's doing a lot. We'll need to improve this for 3.0.1.

It's fairly easy to reproduce I think, by taking any dag, running hundred of dag runs (backfill, or manually), and then trying to display the grid for that dag should display a significant delay. (maybe the more complex the dag is the longer the delay)

[bbovenzi]

Yes you can take any dag with a large amount of tasks and then run it a few hundred times. Even though we paginate to only show 10 runs at a time, oru grid data endpoint is still slowed down.

[zachliu]

Request endpoint: https://my.airflow.com/ui/grid/my_dag?limit=10&order_by=-run_after
Request method: GET
Status Code: 504 Gateway Timeout

1 "dummy" task in the DAG is enough to reproduce this error if you run it a few hundred times.