feat: switch probes to only startupProbe #4861
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
NB: I've tried disabling schema aware validation but nothing changed |
|
3 runs all green except a single failure for the realtime-composition job, which is flapping also on master. So looks like is enough to address the issue we were originally trying to resolve. |
negz
reviewed
Oct 20, 2023
cmd/crossplane/core/core.go
Outdated
| // Add default health probe | ||
| if err := mgr.AddHealthzCheck("ping", healthz.Ping); err != nil { | ||
| return errors.Wrap(err, "cannot create ping health check") | ||
| } |
There was a problem hiding this comment.
Should we leave these, even if we don't use them for now? I could see them being handy in future. Or even just handy so that folks can configure their Helm chart to use them if they want.
There was a problem hiding this comment.
yes, make sense to me, reverted these changes
negz
approved these changes
Oct 20, 2023
Signed-off-by: Philippe Scorsolini <p.scorsolini@gmail.com>
phisco
force-pushed
the
dev/startup-probe-only
branch
from
7415d8e
to
c7bf78f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of your changes
The probes introduced in #4748 could be too aggressive and lead to some undesired restarts under heavier operations, e.g. in my case installing
platform-ref-aws, if the resources are constrained.The original issue we were trying to solve was that we were seeing some failures in the e2es at startup due to the webhooks not being yet ready to receive requests. So, a startupProbe could be enough, although it could lead to the pod being restarted after the
failureThreshold * periodSecondstime window, I've set it to be 60s (30*2s), but we could be even more permissive and increase the failureThreshold if needed. To compensate, I've configured the probe to be just atcpProbe, which should be less affected even if for some reason the pod was under heavier usage.I hit this while testing it locally on the latest commit, by just applying the following manifest:
Seeing a few restarts of the Crossplane pod after that.
With this change I couldn't see any restart, and even killing the crossplane pod while it was trying to deploy
platform-ref-awsdidn't result in any issue at restart. Has to be seen whether it's actually solving the flakiness we were seeing.I have:
make reviewableto ensure this PR is ready for review.Added or updated unit tests.Added or updated e2e tests.Linked a PR or a docs tracking issue to document this change.Addedbackport release-x.ylabels to auto-backport this PR.Need help with this checklist? See the cheat sheet.