Skip to content

Added documentation explaining difference deferred vs async operators#63500

Open
dabla wants to merge 19 commits intoapache:mainfrom
dabla:feature/add-docs-async-python
Open

Added documentation explaining difference deferred vs async operators#63500
dabla wants to merge 19 commits intoapache:mainfrom
dabla:feature/add-docs-async-python

Conversation

@dabla
Copy link
Contributor

@dabla dabla commented Mar 13, 2026

Added documentation explaining difference deferred vs async operators, related to the PR #60268 which added native async support for PythonOperators.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

…async operators with examples and explaining what the difference is
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks okay for me, some proposals as feedback.

@dabla
Copy link
Contributor Author

dabla commented Mar 13, 2026

Thanks for the thorough review @jscheffl will fix the remarks.

Copy link
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doc review of the new "Deferred vs Async Operators" page. The content fills a real gap — users need this guidance. Most comments are about broken cross-references (build failures) and making the examples more complete for users.

@uranusjr
Copy link
Member

I feel we should also discuss when you should not use one of the other. For example

  • You probaly shouldn’t prefer async over defer if the wait can be long (since an async operator is kept running in a process, which can waste resource).
  • You probably shouldn’t use defer if deferring happens many times and are generally short (even if you write a custom deferrable operator) since it stops the process every time it defers and restarts everything it resumes, which can introduce a lot of overhead.

Also, the mention about existing deferrable operator can be a bit misleading IMO. If your use case is better for defer, and there’s no existing deferrable operator available, your best course would be to write one yourself, not to use async. It should depend more about the use case, not what Airflow has available.

@dabla dabla force-pushed the feature/add-docs-async-python branch from ecbd95d to 8638713 Compare March 14, 2026 11:42
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for me now!

Feedback by others welcome though.

@dabla dabla marked this pull request as draft March 16, 2026 18:00
@dabla dabla marked this pull request as ready for review March 16, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants