Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prepare-downscale webhook should check for rollouts #82

Merged
merged 10 commits into from
Sep 21, 2023

Conversation

jhalterman
Copy link
Member

@jhalterman jhalterman commented Sep 14, 2023

This PR updates the prepare-downscale webhook to check for rollouts that are in progress in the rollout-group, by comparing the statefulset's Status.UpdatedReplicas to the number of ready and running pods in the group.

This PR also:

  • Moves some of the util functions that were in the controller package to a separate util package for access by webhooks.
  • Removes the unused min/max functions, since these are built-ins in go 1.21

Fixes #79

@jhalterman jhalterman force-pushed the sts-ready-check branch 2 times, most recently from 4474444 to 9fb9fdc Compare September 14, 2023 19:33
@jhalterman jhalterman marked this pull request as ready for review September 20, 2023 01:36
@jhalterman jhalterman requested a review from a team as a code owner September 20, 2023 01:36
pkg/admission/prep_downscale.go Outdated Show resolved Hide resolved
pkg/admission/prep_downscale.go Outdated Show resolved Hide resolved
pkg/admission/prep_downscale.go Show resolved Hide resolved
pkg/admission/prep_downscale_test.go Show resolved Hide resolved
pkg/admission/prep_downscale.go Outdated Show resolved Hide resolved
pkg/admission/prep_downscale.go Outdated Show resolved Hide resolved
Copy link
Contributor

@56quarters 56quarters left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with some minor feedback. Let's see how it works in a dev environment!

pkg/admission/prep_downscale.go Outdated Show resolved Hide resolved
pkg/admission/prep_downscale.go Outdated Show resolved Hide resolved
@jhalterman
Copy link
Member Author

Tested in dev-01 by attempting to manually downscale a statefulset during a rollout:

kubectl edit sts/ingester-zone-a
error: statefulsets.apps "ingester-zone-a" could not be patched: admission webhook "prepare-downscale-cortex-dev-01.grafana.com" denied the request: downscale of statefulsets/ingester-zone-a in cortex-dev-01 from 6 to 4 replicas is not allowed because statefulset ingester-zone-b has 0 non-updated replicas and 6 non-ready replicas
You can run kubectl replace -f /var/folders/c6/z230vgsx5rb3hd3vtj2bc1ww0000gn/T/kubectl-edit-1712830600.yaml to try this update again.

Then a few minutes later, once the pods were ready:

kubectl edit sts/ingester-zone-a
statefulset.apps/ingester-zone-a edited

@56quarters 56quarters merged commit 634d2ff into grafana:main Sep 21, 2023
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Guard against a concurrent downscaling and rollout
2 participants