Update description on What is Airflow - #70547
Conversation
| `Apache Airflow® <https://github.com/apache/airflow>`_ is an open-source platform for developing, scheduling, | ||
| and monitoring batch-oriented workflows. Airflow's extensible Python framework enables you to build workflows | ||
| connecting with virtually any technology. A web-based UI helps you visualize, manage, and debug your workflows. | ||
| and monitoring workflows — from traditional batch-oriented data pipelines to machine learning, model training, |
There was a problem hiding this comment.
| and monitoring workflows — from traditional batch-oriented data pipelines to machine learning, model training, | |
| and monitoring workflows — from traditional batch-oriented and event-based data pipelines to machine learning, model training, |
With the message queue feature, assets, etc., this is also quite an important feature, I would say.
There was a problem hiding this comment.
event-based is not mature in airflow , we can't custom the batching and re-consume events
also airflow it's a very heavy processing framework compare to temporal , so it's still very batch oriented
Conclusion I prefer honest/boring readme for OSS
There was a problem hiding this comment.
@Miretpl Maybe we can find a better wording to clarify what it means?
There was a problem hiding this comment.
Event-driven scheduling shipped in 3.0, and has its own docs. The README should reflect what the project supports today. I see teams running it in production with strict SLAs, so I wouldn't call the it immature.
There was a problem hiding this comment.
| and monitoring workflows — from traditional batch-oriented data pipelines to machine learning, model training, | |
| and monitoring workflows — from traditional time- or event-triggered batch-oriented data pipelines to machine learning, model training, |
A bit more precise suggestion (let me know what you think). I agree that the previous one was a bit too broad for the current event support in Airflow (broadly speaking).
eladkal
added
backport-to-v3-3-test
9 participants
This PR does not change what we do nor focus. It just clarify that to our community.
This topic has been discussed by the PMCs and we think that it's a good idea to update the description and to clarify that Airflow is very suitable to AI/ML/Agentic workflows.
This PR suggest a possible phrasing for the description and I welcome the community to share their opinions on the wording.
I am marking it with don't-merge label just to give the community enough time to share their thoughts and to make sure we are aligned on the phrasing.
This PR will be open for comments at least till Monday, August 3, 2026 at 08:00 PM UTC., after which it will be merged with the accepted phrasing.