Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore unavailable deployments and handle kubeconfig secret rotation #41

Merged
merged 5 commits into from
Mar 23, 2022

Conversation

ashwani2k
Copy link
Contributor

@ashwani2k ashwani2k commented Mar 7, 2022

What this PR does / why we need it:
This PR handles skipping the scaling operation for deployment when they are not available in the cluster.
The logs are also refined to avoid noise and clutter.
It also handled the bug introduced with reloading of shoot kubeconfig as reported with #36.

Which issue(s) this PR fixes:
Fixes #40 #36

Special notes for your reviewer:

  1. The PR upgrades GO dependency to 1.17
  2. There is also a delay introduced before checking dependents for desire availability. This is introduced as checking for availability immediately after scaling operation always failed as it takes some time for the deployment to become ready.
    Currently it is a hacky way with 2 sec delay as per what was observed from the tests.
    Without this delay also the scaling operation happens but it is honored in the next reconciliation cycle. This creates an overall delay of 30s scaling delay per deployment requiring scaling.
  3. In order to handle the kubeconfig rotation race condition where for some edge case the probe fails to access the latest kubeconfig and continues to fail unless restated. A retry mechanism is introduced which shall update the secret. The retry only acts if the error api errors of
  4. Refactors the error handling for probe.
  5. Updated alpine image to 3.15

Release note:

A bug is fixed which allowed dependency-watchdog to not ignore scaling operations on deployment which are not enabled/deployed in a given cluster
A bug with uploading of a rotated dependency-watchdog-probe secrets is now fixed by refreshing the clients with updated secrets. 

@ashwani2k ashwani2k requested a review from a team as a code owner March 7, 2022 15:36
@gardener-robot gardener-robot added needs/review size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 7, 2022
@gardener-robot gardener-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 19, 2022
@ashwani2k ashwani2k changed the title Handled behaviour for deployments when not available in the cluster Ignore unavailable deployments and handle kubeconfig secret rotation Mar 19, 2022
@ashwani2k
Copy link
Contributor Author

@unmarshall -- As suggested by you, I've introduced a retry mechanism to handle the kubeconfig rotation scenario. Also refined the logs overall and made them leaner for happy path. Moved some of the logs to Level 5.
Kindly have a look at the commit 38437b4

Copy link
Contributor

@unmarshall unmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comments

pkg/scaler/prober.go Outdated Show resolved Hide resolved
pkg/scaler/prober.go Show resolved Hide resolved
pkg/scaler/scaler.go Show resolved Hide resolved
@ashwani2k
Copy link
Contributor Author

Thanks Madhav, I agree we can make a total separation of concerns.
The changes are made with the commit 15f476a and it also tested on a local seed/shoot setup.

@ashwani2k
Copy link
Contributor Author

@unmarshall
Copy link
Contributor

/lgtm

@ashwani2k ashwani2k merged commit 8965732 into gardener:master Mar 23, 2022
@ashwani2k ashwani2k deleted the scale-autoscaler branch March 23, 2022 16:34
unmarshall added a commit that referenced this pull request Mar 28, 2022
…in-rel-0.7.0

[rel-0.7.0] Automated cherry pick of #41: Ignore unavailable deployments and handle kubeconfig secret rotation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
6 participants