Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix overscaling when the controller is much faster then the listener #3371

Merged
merged 6 commits into from Mar 20, 2024

Conversation

nikola-jokic
Copy link
Member

When the ephemeral runner finishes, the listener issues a scale-down patch to the ephemeral runner set. But if ephemeral runner set reacts quicker than the listener, it may re-create the ephemeral runner, and quickly remove it.
To avoid this problem, this PR introduces the following changes:

  1. Ephemeral runner cleans up its resources as soon as it is done
  2. Ephemeral runner set keeps finished ephemeral runners until the next patch request is received.
  3. The target count is calculated only on patch ID difference (or patchID == 0), i.e. when the listener patches the ephemeral runner set.
  4. The listener issues a patch request when there is a change in the state, or in case it gets notified that the job is done.

This change also requires a change in the CRD.

@nikola-jokic nikola-jokic added the gha-runner-scale-set Related to the gha-runner-scale-set mode label Mar 20, 2024
rentziass
rentziass previously approved these changes Mar 20, 2024
Copy link
Member

@rentziass rentziass left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, I just left a bunch of suggestions over a type!

Co-authored-by: Francesco Renzi <rentziass@gmail.com>
@nikola-jokic nikola-jokic merged commit 7a643a5 into master Mar 20, 2024
17 checks passed
@nikola-jokic nikola-jokic deleted the nikola-jokic/ephemeral-runner-set-patch-id branch March 20, 2024 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
gha-runner-scale-set Related to the gha-runner-scale-set mode
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants