Skip to content

Add workers.celery.topologySpreadConstraints & workers.kubernetes.topologySpreadConstraints#64980

Merged
jscheffl merged 3 commits intoapache:mainfrom
Miretpl:add-workers-celery-k8s-spread-constraints
Apr 11, 2026
Merged

Add workers.celery.topologySpreadConstraints & workers.kubernetes.topologySpreadConstraints#64980
jscheffl merged 3 commits intoapache:mainfrom
Miretpl:add-workers-celery-k8s-spread-constraints

Conversation

@Miretpl
Copy link
Copy Markdown
Contributor

@Miretpl Miretpl commented Apr 9, 2026

related: #28880

This PR introduces two new fields: workers.celery.topologySpreadConstraints and workers.kubernetes.topologySpreadConstraints. The workers.topologySpreadConstraints field is now deprecated.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg bot added the area:helm-chart Airflow Helm Chart label Apr 9, 2026
@kaxil kaxil requested a review from Copilot April 10, 2026 19:55
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds executor-specific topology spread constraint configuration for workers in the Helm chart, deprecating the previous shared workers.topologySpreadConstraints field.

Changes:

  • Introduces workers.celery.topologySpreadConstraints and workers.kubernetes.topologySpreadConstraints in values + schema.
  • Adds deprecation messaging for workers.topologySpreadConstraints (schema description + Helm NOTES).
  • Updates Helm tests to validate precedence/overrides for the new fields.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
helm-tests/tests/helm_tests/airflow_core/test_worker_sets.py Extends test matrix to cover Celery worker-set overrides for topology spread constraints.
helm-tests/tests/helm_tests/airflow_core/test_worker.py Parametrizes worker deployment test to validate deprecated vs new Celery-specific topology spread constraints precedence.
helm-tests/tests/helm_tests/airflow_aux/test_pod_template_file.py Parametrizes pod-template-file test to validate deprecated vs new Kubernetes-specific topology spread constraints precedence.
chart/values.yaml Documents deprecation and adds new values keys for Celery/Kubernetes worker topology spread constraints.
chart/values.schema.json Deprecates old schema description and adds schema entries for new keys.
chart/templates/NOTES.txt Adds a deprecation warning when old workers.topologySpreadConstraints is used.
chart/files/pod-template-file.kubernetes-helm-yaml Switches pod template to prefer workers.kubernetes.topologySpreadConstraints over deprecated/shared fields.

@Miretpl Miretpl force-pushed the add-workers-celery-k8s-spread-constraints branch from afae5a9 to 19120ff Compare April 11, 2026 09:16
@Miretpl Miretpl marked this pull request as ready for review April 11, 2026 09:57
@jscheffl jscheffl added the backport-to-chart/v1-2x-test Automatic backport to chart 1.2x maintenance branch label Apr 11, 2026
@jscheffl jscheffl merged commit 511af0a into apache:main Apr 11, 2026
173 of 184 checks passed
github-actions bot pushed a commit that referenced this pull request Apr 11, 2026
…kers.kubernetes.topologySpreadConstraints (#64980)

* Add workers.celery.topologySpreadConstraints & workers.kubernetes.topologySpreadConstraints

* Add newsfragment
(cherry picked from commit 511af0a)

Co-authored-by: Przemysław Mirowski <17602603+Miretpl@users.noreply.github.com>
@github-actions
Copy link
Copy Markdown

Backport successfully created: chart/v1-2x-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
chart/v1-2x-test PR Link

@Miretpl Miretpl deleted the add-workers-celery-k8s-spread-constraints branch April 11, 2026 11:31
jscheffl pushed a commit that referenced this pull request Apr 11, 2026
…kers.kubernetes.topologySpreadConstraints (#64980) (#65055)

* Add workers.celery.topologySpreadConstraints & workers.kubernetes.topologySpreadConstraints

* Add newsfragment
(cherry picked from commit 511af0a)

Co-authored-by: Przemysław Mirowski <17602603+Miretpl@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:helm-chart Airflow Helm Chart backport-to-chart/v1-2x-test Automatic backport to chart 1.2x maintenance branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants