Skip to content

Port ScheduledOverrides to AutoscalingRunnerSet #3564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

asafhm
Copy link

@asafhm asafhm commented Jun 2, 2024

Fixes #3313 and #2986 (closed with no resolution)

Porting the great work by @mumoshu of the ScheduledOverrides capability of HorizontalRunnerAutoscaler to AutoscalingRunnerSet

How to use it

It's meant to be used in an almost identical way, with one exception - minReplicas was renamed to minRunners to conform with the current naming choice.

Examples:

  1. Within the AutoscalingRunnerSet CR directly:
apiVersion: actions.github.com/v1alpha1
kind: AutoscalingRunnerSet
metadata:
  annotations: # abbreviated
  labels: # abbreviated
  name: example-gha-runner-scale-set
spec:
  githubConfigSecret: "<some_value>"
  githubConfigUrl: "<some_value>"
  maxRunners: 2
  minRunners: 1
  scheduledOverrides:
  - startTime: "2021-05-01T00:00:00+09:00"
    endTime: "2021-05-03T00:00:00+09:00"
    recurrenceRule:
      frequency: Weekly
      untilTime: "2024-07-01T00:00:00+09:00"
    minRunners: 0
  1. With the gha-runner-scale-set helm chart, by setting scheduledOverrides in the values:
minRunners: 1
maxRunners: 2

scheduledOverrides:
  - startTime: "2021-05-01T00:00:00+09:00"
    endTime: "2021-05-03T00:00:00+09:00"
    recurrenceRule:
      frequency: Weekly
      untilTime: "2024-07-01T00:00:00+09:00"
    minRunners: 0

Notable changes

Changes to the AutoscalingRunnerSet Status

  1. Like the previous implementation of scheduled overrides, this PR introduces the ScheduledOverridesSummary field in the AutoscalingRunnerSet Status.

  2. In order to share the desired minimum runners between the AutoscalingRunnerSet and AutoscalingListener resources post scheduled overrides evaluation, an additional field is introduced to the AutoscalingRunnerSet Status - DesiredMinRunners.
    Whenever the listener's minRunners value is different than DesiredMinRunners, the listener is deleted so it can be recreated.

Changes to requeuing of requests

In order to reevaluate the scheduled overrides, the final return ctrl.Result{}, nil in the autoscalingrunnerset reconciliation function was changed to requeue the request after 1 minute.

@kahirokunn
Copy link
Contributor

👏

@gfrid
Copy link

gfrid commented Jun 5, 2024

bump

@asafhm asafhm force-pushed the port-scheduleoverrides-to-autoscalingrunnersets branch from 62269e4 to fdf0aee Compare June 8, 2024 20:56
@asafhm
Copy link
Author

asafhm commented Jun 9, 2024

Unlike the legacy actions-runner-controller, periodic reconciliation of resources is practically disabled in the new controller due to WithEventFilter(predicate.ResourceVersionChangedPredicate{}).
To know if a scheduled override should take place, periodic reconciliation must occur for the AutoscalingRunnerSet resource in one way or another.
In order to make it work here, I used Result.RequeueAfter, and it indeed worked fine, but I believe that going with a standard sync period could eliminate potential scenarios of missing an override due to a bug in the reconciliation logic.

So a small question for the maintainers -
What was the reason behind ignoring the resync period in the gha-runner-scale-set-controller? And if the concern is overloading the controller when many resources exist, won't it make sense to enable it just for the AutoscalingRunnerSet resource?
I'd love to know.

@asafhm asafhm changed the title Port ScheduleOverrides to AutoscalingRunnerSet Port ScheduledOverrides to AutoscalingRunnerSet Jun 10, 2024
@asafhm asafhm force-pushed the port-scheduleoverrides-to-autoscalingrunnersets branch from fdf0aee to a8b1b0e Compare June 16, 2024 12:23
@asafhm
Copy link
Author

asafhm commented Jun 26, 2024

Hi @nikola-jokic
I can see you're the most active code owner here. Is there anything else needed to push this forward?

@XciD
Copy link

XciD commented Jul 22, 2024

Bump on this one, really needed feature

@regmontanhani
Copy link

Hey team, do we have an ETA on this? P+ customer Itaú is asking for a status.

@tkdwill
Copy link

tkdwill commented Jul 31, 2024

Hi Team, any update on this? We met with the customer today.

@caiocsgomes
Copy link

It would be nice to have a position on this. If it is going to be merged or not. This way we could define what to do to tackle this issue in our clusters.

@pragmaticivan
Copy link

👋🏽 Any chance we get any sort of feedback on this PR? This is a very important feature for cost management

@pragmaticivan
Copy link

pragmaticivan commented Nov 13, 2024

@asafhm Thoughts on also supporting cronOverrides with:

apiVersion: actions.github.com/v1alpha1
kind: AutoscalingRunnerSet
metadata:
  annotations: # abbreviated
  labels: # abbreviated
  name: example-gha-runner-scale-set
spec:
  githubConfigSecret: "<some_value>"
  githubConfigUrl: "<some_value>"
  maxRunners: 2
  minRunners: 1
  cronOverrides:
  -  timezone: America/Chicago
      start: 0 6 * * *        # At 6:00 AM
      end: 0 20 * * *        # At 8:00 PM
      minRunners: 0

My main usecase is to have 0 minRunners over the weekend for example.

@asafhm
Copy link
Author

asafhm commented Nov 22, 2024

@asafhm Thoughts on also supporting cronOverrides with:

apiVersion: actions.github.com/v1alpha1
kind: AutoscalingRunnerSet
metadata:
  annotations: # abbreviated
  labels: # abbreviated
  name: example-gha-runner-scale-set
spec:
  githubConfigSecret: "<some_value>"
  githubConfigUrl: "<some_value>"
  maxRunners: 2
  minRunners: 1
  cronOverrides:
  -  timezone: America/Chicago
      start: 0 6 * * *        # At 6:00 AM
      end: 0 20 * * *        # At 8:00 PM
      minRunners: 0

My main usecase is to have 0 minRunners over the weekend for example.

@pragmaticivan You could achieve this with scheduledOverrides too, like this:

  # Scale down on weekends (Friday 6pm - Monday 8am America/Chicago time)
  - startTime: "2024-11-22T18:00:00-06:00"
    endTime: "2024-11-25T08:00:00-06:00"
    recurrenceRule:
      frequency: Weekly
    minRunners: 0

The advantage here is that you can be specific about the future starting point of this rule up to the date itself, which is impossible using Cron, since it will start from the next match of its pattern. I suppose there may be use cases that require specificity for some.

@asafhm
Copy link
Author

asafhm commented Nov 22, 2024

Honestly, it appears that this repository in its entirety is not actively maintained anymore, and for the past two months all commits came from bots, regardless of the large amount of PRs waiting to be looked at.
That is unfortunate, given the obvious need here for this feature, and the popularity of self-hosted runners on Kubernetes in itself...

@kahirokunn
Copy link
Contributor

I know that many people take their holidays in September and October, so I guess there's no way around it, but they don't come back very quickly. 😢

@asafhm
Copy link
Author

asafhm commented Nov 24, 2024

I know that many people take their holidays in September and October, so I guess there's no way around it, but they don't come back very quickly. 😢

Sadly this trend began before September, and even before this PR was created :/

@asafhm
Copy link
Author

asafhm commented Jan 1, 2025

@Link- Trying my luck here since no other maintainer appears active recently... Could I kindly solicit a review? There's enough demand from the community for this PR

@IdanYaffe
Copy link

Bump

@asafhm
Copy link
Author

asafhm commented Feb 13, 2025

I've given up hope this or any other long-living PR by the community will eventually get merged.
For those here who still need a workaround, there's an operator named kube-green that seems nice and may help. It doesn't appear to be too complex and doesn't support scenarios of specific dates for scaling down, but if your use case is to scale down your runners on weekends/out-of-office hours, you should be fine.
I haven't tried it myself since my situation with ARC changed over the last several weeks, but you can use its custom patches capability to support Custom Resources such as AutoscalingRunnerSet.

@ali-kafel
Copy link

@nikola-jokic Would it be possible to get a review on this please?

@caiocsgomes
Copy link

Hey team! Any reason why this was not merged in the release from today? It would be nice to have at least a position on this, an approval or rejection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature Request: Schedule-based minRunners Configuration
10 participants