Create backoff mechanism for failed runners and allow re-creation of failed ephemeral runners #4059
Conversation
…failed ephemeral runners
nikola-jokic
requested review from
mumoshu,
toast-gear,
rentziass and
a team
as code owners
nikola-jokic
added
the
gha-runner-scale-set
|
This would be a big help to us. We're having some issues with our Kubevirt runners that we need to resolve, but this would help us out until we can nail down the precise problem. |
|
Could we please prioritize this PR? We are currently experiencing the same problem as described in #2721, and it is impacting our workflow. |
|
Hey @rr-krupesh-savaliya, I just want to say that depending on the root cause of the failures happening on your cluster, you might still want to inspect them and fix them. This is just a measure that would allow self-healing, but underlying problems should still be fixed, so you can get all the benefits of autoscaling |
| } | ||
| return updated.Status.Reason, nil | ||
| }, ephemeralRunnerTimeout, ephemeralRunnerInterval).Should(BeEquivalentTo("TooManyPodFailures"), "Reason should be TooManyPodFailures") | ||
| for i := range 5 { |
There was a problem hiding this comment.
Do we have coverage for partial failures? Do we want to add one more test case to cover behaviour for less than < 5 failures?
There was a problem hiding this comment.
There is a test that covers it on eviction, which kind of covers the case for re-creation, and we have the case that checks the exit code 0 when the runner exists in the service. I feel like these tests are covering the less than 5 failures.
Fixes #2721
Fixes #3685
Fixes #3700