Skip to content

Move task execution architecture docs to contributing-docs - #70657

Open
jason810496 wants to merge 1 commit into
apache:mainfrom
jason810496:docs/go-sdk/coordinator-first-in-overview
Open

Move task execution architecture docs to contributing-docs#70657
jason810496 wants to merge 1 commit into
apache:mainfrom
jason810496:docs/go-sdk/coordinator-first-in-overview

Conversation

@jason810496

@jason810496 jason810496 commented Jul 29, 2026

Copy link
Copy Markdown
Member

Why

The "Task execution architecture" section added in #69750 documents worker internals (the Supervisor, the Coordinator layer, and the language subprocess protocols). That is developer-facing material, not something end users need in the core-concepts/overview.rst page, so it belongs in the contributing docs instead.

How

  • Move the whole section (Python Task SDK execution, the Coordinator path shared by the Java and Go SDKs, and the standalone Go edge worker) into a new dev-facing page contributing-docs/31_task_execution_architecture.rst, next to the new-language-SDK guide.
  • Move the five diagrams to contributing-docs/images/ and extend the generate-airflow-diagrams prek hook to cover that directory. Rewrite the cross-links for GitHub rendering (relative .rst links, .. image:: images/...).
  • The two former Java-only diagrams are generalized into shared Coordinator diagrams (Java + Go); the native-language diagram is renamed to the Go edge worker it represents.
  • Drop the section from core-concepts/overview.rst; register the new page in the contributing-docs README and cross-link it from 30_new_language_sdk.rst.

Was generative AI tooling used to co-author this PR?

@jason810496 jason810496 self-assigned this Jul 29, 2026
@jason810496
jason810496 marked this pull request as ready for review July 29, 2026 08:09
@jason810496
jason810496 requested a review from potiuk July 29, 2026 08:09

@uranusjr uranusjr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t think we want to put coordinator first. I intentionally put the concept to the back because the user does not need to care how it works, only how to use it. The new writeup emphasis on internals too much. It should instead focus on introducing how the coordinator can be used (by configuration) instead.

@jason810496

Copy link
Copy Markdown
Member Author

I don’t think we want to put coordinator first. I intentionally put the concept to the back because the user does not need to care how it works, only how to use it. The new writeup emphasis on internals too much. It should instead focus on introducing how the coordinator can be used (by configuration) instead.

Hmmm, then I will need to move the whole paragraph introduce in #69750 to somewhere else.

All the diagrams and descriptions are quite verbose for the users IMO -- http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/core-concepts/overview.html#non-python-language-sdks-go-and-java

@jason810496

jason810496 commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

Then would it be better to move them all (starting from "Task execution architecture" section) to airflow-core/docs/core-concepts/task-execution-archciture.rst as a new page instead of overwhelming the core-concepts/overview.rst page?

@uranusjr

uranusjr commented Jul 29, 2026

Copy link
Copy Markdown
Member

I think they do not belong in user-facing documentation. It would fit if we have a section for advanced concepts or something, but we currently don’t. They do fit in dev-facing docs though (contributing-docs).

The "Task execution architecture" section added in apache#69750 documents
worker internals (the Supervisor, the Coordinator layer, and the language
subprocess protocols). That is developer-facing material, not something
end users need in the core-concepts overview. Relocate the whole section
and its diagrams to a new dev-facing page under contributing-docs, next
to the new-language-SDK guide, and drop it from the user-facing overview.

Follow-up to review feedback on apache#70657.
@jason810496
jason810496 force-pushed the docs/go-sdk/coordinator-first-in-overview branch from 8271810 to bf8246c Compare July 29, 2026 09:31
@jason810496 jason810496 changed the title Put the Coordinator path first in the non-Python SDK docs Move task execution architecture docs to contributing-docs Jul 29, 2026
@jason810496

Copy link
Copy Markdown
Member Author

I just moved them all to contributing-docs/31_task_execution_architecture.rst (a new page under contributing-docs that don't make the new task-sdk one too large).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants