Skip to content

Add option to override Celery worker images#63514

Open
sg-c0de wants to merge 1 commit intoapache:mainfrom
sg-c0de:celery-worker-image-override
Open

Add option to override Celery worker images#63514
sg-c0de wants to merge 1 commit intoapache:mainfrom
sg-c0de:celery-worker-image-override

Conversation

@sg-c0de
Copy link

@sg-c0de sg-c0de commented Mar 13, 2026

Summary

Since we have the option to configure multiple Celery worker sets, it is useful to be able to use different images for specific sets (for example, to include CUDA drivers). This adds the option to override the image for all Celery workers globally or on a per-set basis using Values.workers.celery.image and Values.workers.celery.sets[].image.

Changes

chart/templates/_helpers.yaml: Add airflow_worker_image, airflow_worker_image_pull_policy, and airflow_worker_image_for_migrations helpers, and patch the fullOverwrite helper to support image overrides.

chart/templates/workers/worker-deployment.yaml: Update image and imagePullPolicy references to use the new helper templates.

chart/values.yaml: Add new values to support the image overrides.

@boring-cyborg
Copy link

boring-cyborg bot commented Mar 13, 2026

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@boring-cyborg boring-cyborg bot added the area:helm-chart Airflow Helm Chart label Mar 13, 2026
Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that option makes sense to me. Can you also please add some tests?

@potiuk
Copy link
Member

potiuk commented Mar 16, 2026

@sg-c0de This PR has a few issues that need to be addressed before it can be reviewed — please see our Pull Request quality criteria.

Issues found:

  • ⚠️ Missing Tests: The PR adds new Helm chart behavior (Celery worker image overrides via new helper templates and new values) but no Helm unit test files are included in the changeset. Per review guidelines, new public behavior requires tests covering success, failure, and edge cases. The Helm tests suite has a dedicated area for exactly this kind of templating logic.
  • ⚠️ Description Does Not Match Code: The PR description states the new values are at Values.workers.celery.image and Values.workers.celery.sets[].image, but the _helpers.yaml patch references .Values.workers.image (not .Values.workers.celery.image). The values.yaml patch adds the image block inside the workers.celery section (after queue: "default"). This discrepancy between the described API surface and the actual template code is a code-correctness concern that warrants scrutiny from maintainers.

Note: Your branch is 73 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

  • The comment informs you what you need to do.
  • Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
  • There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
  • Maintainers will then proceed with a normal review.

There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:helm-chart Airflow Helm Chart

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants