Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable native HPA for Airflow Workers #36174

Merged
merged 13 commits into from Jan 14, 2024

Conversation

pemiranda-clgx
Copy link
Contributor

@pemiranda-clgx pemiranda-clgx commented Dec 11, 2023

Add K8S native Horizontal Pod Autoscaler in Airflow Workers to support autoscaling based on external custom metrics.

Copy link
Member

@hussein-awala hussein-awala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good; I just added a small comment about the chosen type for the HPA metrics.

chart/values.schema.json Outdated Show resolved Hide resolved
chart/values.yaml Outdated Show resolved Hide resolved
chart/templates/workers/worker-hpa.yaml Show resolved Hide resolved
chart/values.yaml Outdated Show resolved Hide resolved
helm_tests/other/test_hpa.py Show resolved Hide resolved
helm_tests/other/test_hpa.py Show resolved Hide resolved
Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the changes @pemiranda-clgx
I have one concern regarding versioning.

It should be /v2 for K8S >= 1.23 and /v2beta2 for < 1.23

chart/templates/workers/worker-hpa.yaml Show resolved Hide resolved
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@potiuk potiuk merged commit 03cf692 into apache:main Jan 14, 2024
59 checks passed
@ephraimbuddy ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Feb 19, 2024
abhishekbhakat pushed a commit to abhishekbhakat/my_airflow that referenced this pull request Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:helm-chart Airflow Helm Chart changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants