API - UI Optimization

[pierrejeambrun]

This is the meta issue for tracking performance optimization of the UI and API to improve airflow operating at high scale.

I - API

• Add gunicorn support for API server with rolling worker restarts #60940 (memory optimization and worker rotation)
• API - Improve historical_metrics_data endpoint #62021
• API - Improve public get_dag_runs endpoint #62025
• API - Improve public get_task_instances endpoint #62027
• Mention http2 in the API/UI performance documentation #62032
• Optimize /ui/dags endpoint serialization #61483
• requires_access_dag checks made per grid ti summary API call rendering the dag page are redundant #61485 Cache dag access and teams check to be reused in grid ti_summary API calls #61623

II - UI

• Add virtualization to grid view #60241
• Optimize grid node merging with ID index #61656
• UI - Improve refresh rate strategy of the Grid view #62033
• Replace node-sql-parser with sqlparser-ts #61111

Attached is a zip file with some dummy dags that will generate a high number of tasks and runs in your local development setup. This is useful to 'simulate' a high scale installation with millions of TIs and thousands of dag runs, to help you work on the linked issues or identify other bottlenecks.

serdagexpensive.zip

[tirkarthi]

Thanks for the meta issue. I have been also profiling and testing our performance in the internal environment. For learning purposes I have created a repo to just implement the latest dag_run endpoint in Go, Starlette and FastAPI skipping auth and dag authorization checks. I have found that directly using async driver like aiopg/aiomysql to skip SQLAlchemy usage but still use pydantic delivers up to 2x improvement. Go is up to 10x faster with low resource usage which is not surprising .

Another issue to look is around performance of FastAPI under no load and high concurrent load. Loading a dag detail page with 10 dagruns makes close to 40 API calls at the same including 1 API call per dagrun in the grid. As opposed to Airflow 2 where only 4 calls were made for the grid object data which was split in Airflow 3 to become an API call per dagrun in the grid. Under initial load of dag page the API time is high and once the UI loading is done as backend is free re-running the same API call from Chrome/Firefox one at a time is much faster. So there needs to be some analysis on the effects of concurrent requests being made from UI on loading a dag page.

Repo : https://github.com/tirkarthi/airflow-go

[pierrejeambrun]

Thanks @tirkarthi for the details, feel free to create and add issues as you see improvement opportunities. (rewriting the API in Go I'm not sure though 😂).

Under initial load of dag page the API time is high and once the UI loading is done as backend is free re-running the same API call from Chrome/Firefox one at a time is much faster

#62032 This might answer that.