RunnerSet Cannot Deploy to Different Namespaces in the same Cluster

[rxa313]

Unlike RunnerDeployments, RunnerSets cannot be deployed in the same Cluster even while being in a different Namespace.

I was previously able to successfully deploy RunnerDeployments to the same Clusters all with their own controllers and had no issues with them having shared values (different helm release names - see 782).. Now, I'm deploying RunnerSet and am seeing an issue where the API call is being shared from a separate namespace, even with watchNamespace set to the namespace of the controller.

Currently running helm chart version 0.13.2 and actions-runner-controller version 0.20.2

To reproduce, create a namespace, deploy a controller with a unique release name, deploy a RunnerSet to that controller.
Create another namespace, deploy a controller with a unique release name AND a unique GitHub App installation ID, OR (for my case) a PAT for GitHub Enterprise Cloud/GitHub Enterprise Server (whichever you want, in my case I have two RunnerSets for GHES for two separate organizations using the same PAT, when I try to deploy to our organization in GHEC and use a new PAT there for that instance, I am seeing the error:

create Pod xx-ghec-runnerset-0 in StatefulSet xx-ghec-runnerset failed error: admission webhook "mutate-runner-pod.webhook.actions.summerwind.dev" denied the request: failed to create registration token: POST https://github.xxx.com/api/v3/orgs/xxx/actions/runners/registration-token: 404 Not Found []

^ the API call above should be pointed to github.com, but instead is pointed to my GHES URL, when in my values.yml I did not specify any GitHub Enterprise Server URL:

env:
  GITHUB_ENTERPRISE_URL: ""

I expect to be able to deploy RunnerSet the same way we can deploy RunnerDeployments to different Namespaces within the same Cluster.

I'm moving to RunnerSets for now as it has the ability to accept dnsPolicy: Default which solves timeout errors we were seeing when deploying to our Kubernetes cluster. Would like to go back to RunnerDeployments if the dnsPolicy behavior becomes available there as we want to utilize autoscaling.

If this functionality will not be supported for RunnerSets and you guys recommend to only use RunnerDeployments please let me know.
Thanks!

[mumoshu]

@rxa313 Thanks for reporting! I have never thought about this but in my theory, this happens due to the admission webhook server from each unique installation of actions-runner-controller are interfering with each other.

We use the admission webhook server to inject github tokens to runner pods managed by StatefulSet managed by RunnerSet. RunnerDeployment doesn't need an admission webhook server to inject tokens to runner pods as we have full control over how we create pods in that case. So, that's the difference.

Probably we need to somehow tweak our admission webhook server code:

https://github.com/actions-runner-controller/actions-runner-controller/blob/8657a34f3275f72f67b4c21ce6482e444d087c55/main.go#L255-L259

Or tweak our webhook config so that each admission webhook is involved only when the target pod is in the desired namespace.

https://github.com/actions-runner-controller/actions-runner-controller/blob/master/charts/actions-runner-controller/templates/webhook_configs.yaml

[mumoshu]

@rxa313 I have some good news. There's namespaceSelector in an admission webhook configuration that allows you to restrict that webhook to be involved only in certain namespaces.

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector

The below is the config for the webhook that intercepts pods managed by runnersets:

https://github.com/actions-runner-controller/actions-runner-controller/blob/5805e39e1fb9d5b433352bfe7c792782663b7500/charts/actions-runner-controller/templates/webhook_configs.yaml#L71-L92

Given all that, let's say you had two releases one and two, where one has watchNamespace=foo and two has watchNamespace=bar.

You label the namespace foo with a label actions-runner-controller/id: "one", and label bar with actions-runner-controller/id: "two"`.

kind: Namespace
metadata:
  name: foo
  labels:
    actions-runner-controller/id: "one"
---
kind: Namespace
metadata:
  name: bar
  labels:
    actions-runner-controller/id: "two"

In one's webhook config you set namespaceSelector like:

  name: mutate-runner-pod.webhook.actions.summerwind.dev
  namespaceSelector:
    matchExpressions:
    - key: actions-runner-controller/id
      operator: In
      values: ["one"]
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods

In two's webhook config you set it like:

  name: mutate-runner-pod.webhook.actions.summerwind.dev
  namespaceSelector:
    matchExpressions:
    - key: actions-runner-controller/id
      operator: In
      values: ["two"]
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods

This way, ones webhook tries to mutate runnerset pods in foo only, where twos webhook tries to mutate pods in bar only.

[rxa313]

@mumoshu

Thanks for the information - this is very interesting.

I want to confirm that this worked for me - it's a great feature for those of us who have both GHES and GHEC. I now have a namespace with a RunnerSet for GHEC and another namespace with RunnerSets for my GHES organizations and no conflicts so far.

I also am just curious, Autoscaling is not a feature of RunnerSet or is it available as well? I tried implementing but didn't see any autoscaling with the RunnerSet.

[mumoshu]

@rxa313 Thanks for your confirmation! Glad to hear it worked for you ☺️ And also thanks for sharing your experience. It would help everyone who will try the same setup in the future.

I also am just curious, Autoscaling is not a feature of RunnerSet or is it available as well? I tried implementing but didn't see any autoscaling with the RunnerSet.

How did you configure scale up triggers?

I may have missed documenting it, but RunnerSet supports autoscale by new workflow_job webhook events only. Please see https://github.com/actions-runner-controller/actions-runner-controller#example-1-scale-on-each-workflow_job-event for how you could set up workflow_job based autoscale.

Do you need support for events other than workflow_job for RunnerSet? If so, may I ask why?

Also, which runner kinds are you using, repository, organizational, or enterprise runners?

I recently realized that we have no implementation for enterprise runner autoscale at all. It has been implemented in #906 and verified by @roshvin at #892 (comment).

So If you're trying to autoscale enterprise runners, it would be super helpful if you could give it a try by building a custom build of actions-runner-controller and deploying it onto your cluster. The process described in #908 would be helpful for that.

[rxa313]

Hi @mumoshu

Thanks for the info. We're not configuring autoscaling at the enterprise level, only at our organization level at this time.

When setting up the workflow_job webhook events, and the webhook configuration. Is there some payload_url that is supposed to be reached and set-up in my github instance?

I configured the install of the github webhook in my values.yaml file, but only changed enabled: true:

githubWebhookServer:
  enabled: true
  replicaCount: 1
  syncPeriod: 1m
  secret:
    create: true
    name: "github-webhook-server"
....

Is there some further configuration that is needed to setup the webhook? In my runnerset yaml file I configured this as well:

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
   name: dev-runner-auto-scaler-deployment
spec:
  minReplicas: 1
  maxReplicas: 20
  scaleTargetRef:
    name: dev-corp-org-runner-deployment
  scaleUpTriggers:
  - githubEvent: {}
    duration: "3m"

I applied the changes and see the Workload: actions-runner-controller-test-github-webhook-server is deployed in rancher, but when I run a workflow with say 4 jobs in the queue, nothing is scaled.

Appreciate your help.

Update 2
I removed all runnersets and HA configurations and deployed a RunnerDeployment to my namespace. I notice the same thing is happening here as well, nothing is autoscaling even after the webhook server is installed. I'm sure something is not being connected properly.

also my controller logs don't look like the logs I've see from this issue. I don't see any "event:", "workflow_job" in my manager logs

[mumoshu]

@rxa313 Hey! I guess you missed exposing the webhook autoscaler's HTTP server via a loadbalancer or an ingress, and/or registering the webhook server's URL on GitHub UI.

[rxa313]

@mumoshu You're right, that is what I missed.

It might be worth documenting that in the values.yml file you must edit enabled: true, service: type: NodePort, nodePort: xxx

githubWebhookServer:
  enabled: true
  replicaCount: 1
  syncPeriod: 1m
  secret:
    create: true
    name: "github-webhook-server"
    ### GitHub Webhook Configuration
    #github_webhook_secret_token: ""
  imagePullSecrets: []
  nameOverride: ""
  fullnameOverride: ""
  serviceAccount:
    # Specifies whether a service account should be created
    create: true
    # Annotations to add to the service account
    annotations: {}
    # The name of the service account to use.
    # If not set and create is true, a name is generated using the fullname template
    name: ""
  podAnnotations: {}
  podLabels: {}
  podSecurityContext: {}
  # fsGroup: 2000
  securityContext: {}
  resources: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}
  priorityClassName: ""
  service:
    type: NodePort
    ports:
      - port: 80
        targetPort: http
        protocol: TCP
        name: http
        nodePort: xxx
  ingress:
    enabled: false
    annotations:
      {}
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"
    hosts:
      - host: chart-example.local
        paths: []
    tls: []
    #  - secretName: chart-example-tls
    #    hosts:
    #      - chart-example.local

Now that this is connected and I sent ping to server with proper status, I'm unfortunately still getting a response 200 which is good, but the body says:
no horizontalrunnerautoscaler to scale for this github event

I'm seeing in my webhook server debug issues similar to This Comment

2021-10-26T15:06:04.823Z	DEBUG	controllers.Runner	Found 0 HRAs by key	{"key": "xxx"}
2021-10-26T15:06:04.823Z	DEBUG	controllers.Runner	Found 0 HRAs by key	{"key": "xxx"}
2021-10-26T15:06:04.823Z	DEBUG	controllers.Runner	no repository runner or organizational runner found	{"event": "workflow_job", "hookID": "xxx", "delivery": "xxxx", "workflowJob.status": "completed", "workflowJob.labels": [], "repository.name": "xxx", "repository.owner.login": "xxx", "repository.owner.type": "Organization", "action": "completed", "repository": "xxx/xxx", "organization": "xx-CORP_IT"}

I notice in this log is states

Scale target not found. If this is unexpected, ensure that there is exactly one repository-wide or organizational runner deployment that matches this webhook event

I was under the impression that RunnerSet is supported by this, not just RunnerDeployment?

My runnerset.yaml file:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: dev-example-runnerset
spec:
  # MANDATORY because it is based on StatefulSet: Results in a below error when omitted:
  #   missing required field "selector" in dev.summerwind.actions.v1alpha1.RunnerSet.spec
  selector:
    matchLabels:
      app: dev-example-runnerset

  # MANDATORY because it is based on StatefulSet: Results in a below error when omitted:
  # missing required field "serviceName" in dev.summerwind.actions.v1alpha1.RunnerSet.spec]
  serviceName: dev-example-runnerset

  #replicas:

  # From my limited testing, `ephemeral: true` is more reliable.
  # Seomtimes, updating already deployed runners from `ephemeral: false` to `ephemeral: true` seems to
  # result in queued jobs hanging forever.
  ephemeral: true

  organization: XX-CORP-IT
  #
  # Custom runner image
  #
  image: xxx/summerwind/xx-runner_test:5
  #
  # dockerd within runner container
  #
  ## Replace `mumoshu/actions-runner-dind:dev` with your dind image
  #dockerdWithinRunnerContainer: true
  #
  # Set the MTU used by dockerd-managed network interfaces (including docker-build-ubuntu)
  #
  dockerMTU: 1450
  
  #Runner group
  # labels:
  # - "mylabel 1"
  # - "mylabel 2"
  labels:
  - rancher-runner-set
  
  #group: RunnerSetGroupTest
  #
  # Non-standard working directory
  #
  # workDir: "/"
  template:
    metadata:
      labels:
        app: dev-example-runnerset
    spec:
      dnsPolicy: Default
      containers:
      - name: runner
        imagePullPolicy: IfNotPresent
        env:
        - name: NODE_EXTRA_CA_CERTS
          value: /usr/local/share/ca-certificates/xxx.crt
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
   name: dev-example-auto-scaler-runnerset
spec:
  minReplicas: 1
  maxReplicas: 8
  scaleTargetRef:
    name: dev-example-runnerset
  scaleUpTriggers:
  - githubEvent: {}
    duration: "3m"

HRA Desciption:

❯ kubectl describe HorizontalRunnerAutoscaler dev-example-auto-scaler-runnerset -n actions-runner-system
Name:         dev-example-auto-scaler-runnerset
Namespace:    actions-runner-system
Labels:       <none>
Annotations:  <none>
API Version:  actions.summerwind.dev/v1alpha1
Kind:         HorizontalRunnerAutoscaler
Metadata:
  Creation Timestamp:  2021-10-26T12:25:03Z
  Generation:          4
  Managed Fields:
    API Version:  actions.summerwind.dev/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:maxReplicas:
        f:minReplicas:
        f:scaleTargetRef:
          .:
          f:name:
        f:scaleUpTriggers:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-10-26T14:56:39Z
  Resource Version:  681894630
  Self Link:         /apis/actions.summerwind.dev/v1alpha1/namespaces/actions-runner-system/horizontalrunnerautoscalers/dev-example-auto-scaler-runnerset
  UID:               xxx
Spec:
  Max Replicas:  8
  Min Replicas:  1
  Scale Target Ref:
    Name:  dev-example-runnerset
  Scale Up Triggers:
    Duration:  3m
    Github Event:
Events:  <none>

UPDATE @mumoshu
I've deployed a RunnerDeployment and removed the HRA for the RunnerSet and I can see that the RunnerDeployment scales based on the workflow_job webhook just fine. I triggered 4 jobs and 4 additional runners automatically spun up. I believe this confirms that RunnerSet might not support HRA based on github webhook, unless you can see that there is something missing from my above configurations.

[rxa313]

UPDATE 2
@mumoshu

Once again, apologies for the multiple comments on this thread. I've done some more digging, my bad - HRA does indeed support RunnerSet. I found this digging through some issue/merges and came upon this: merge

I was missing a critical step in the HRA configuration: must change kind: RunnerSet <- This is critical:

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
   name: dev-example-auto-scaler-runnerset
spec:
  minReplicas: 1
  maxReplicas: 8
  scaleTargetRef:
    kind: RunnerSet
    name: dev-example-runnerset
  scaleUpTriggers:
  - githubEvent: {}
    duration: "3m"

One question for you, how can we scale down more efficiently? I have it set to duration: 3m but the runners are still existing after 3m have passed.

[mumoshu]

@rxa313 Thanks for testing and sharing your experience!! Regarding your last question, duration: 3m is noop as scale-down happens without a delay controlled by that setting in a case of workflow_job based autoscale.

Perhaps you'd need to configure HRA.spec.scaleDownDelaySecondsAfterScaleUp to a very small value of your choice.

The default value for it is 10 minutes (https://github.com/actions-runner-controller/actions-runner-controller/blob/98da4c2adb64e253197129e701a61d7faa3427d6/controllers/horizontalrunnerautoscaler_controller.go#L496-L500) which basically means that the scale-down on workfow_job of completed is deferred until it passes 10 minutes after the corresponding scale-up triggered before by aworkflow_job of queued.

Make it small as possible, so that scale-down happens as early as possible. But I haven't tried lowering it too much myself as it is for preventing scale up/down flapping. I'd appreciate your test report!

[rxa313]

@mumoshu

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
   name: dev-example-auto-scaler-runnerset
spec:
  scaleDownDelaySecondsAfterScaleUp: 1m
  minReplicas: 1
  maxReplicas: 5
  scaleTargetRef:
    kind: RunnerSet
    name: dev-example-runnerset
  scaleUpTriggers:
  - githubEvent: {}
    duration: 3m

error validating data: ValidationError(HorizontalRunnerAutoscaler.spec): unknown field "scaleDownDelaySecondsAfterScaleUp" in dev.summerwind.actions.v1alpha1.HorizontalRunnerAutoscaler.spec; if you choose to ignore these errors, turn validation off with --validate=false

What am I doing wrong here?

[toast-gear]

https://github.com/actions-runner-controller/actions-runner-controller/blob/b805cfada7863a23946b5ae362d4c071dbaa7a43/config/crd/bases/actions.summerwind.dev_horizontalrunnerautoscalers.yaml#L99-L101 try setting scaleDownDelaySecondsAfterScaleOut instead

[rxa313]

@toast-gear ,

Thanks so much, that was the issue. The former does not work in my instance, but scaleDownDelaySecondsAfterScaleOut: 60 worked.

@mumoshu

In my testing, setting to 60 seconds or 120 seconds scaled down a lot smoother than the default 10m for myself, especially when testing how it works. Scaling down as early as a minute is a good amount of time for my organization, I'll have more testing once it's deployed into production.

Is there some type of delay when kicking off scaling up from 0? I'm sure you've seen issues where scaling up from 0 cause a no runners found with label error appears. How can we trigger without erroring before the runner spins up.

[mumoshu]

@rxa313 Hey! Thanks for testing it out. Apparently I made a typo when implementing scaleDownDelaySecondsAfterScaleOut 😅

Re no runners found with label errors, I believe it happens only on GHES(GitHub Enterprise Server) but GitHub Cloud.

For GitHub Enterprise, you'll probably receive a fix on GitHub Enterprise Server 3.3(I think that's the next version). Until then, you'd need to use RunnerDeployment to avoid no runners found with label error on GHES.

Summary: Scale-from-zero works with...

• GHES 3.2 + RunnerDeployment (thanks to our registration-only runners)
• GH + RunnerDeployment (GH supports scale-from-zero out-of-box since 20 Sep https://github.blog/changelog/2021-09-20-github-actions-ephemeral-self-hosted-runners-new-webhooks-for-auto-scaling/
• GH + RunnerSet (Same as above

Scale-from-zero doesn't work with...

• GHES 3.2 + RunnerSet (GHES 3.2 doesn't support scale-from-zero out-of-box and RunnerSet doesn't support registration-only runners

[toast-gear]

@rxa313 you'll know if 3.3 has the fix in because the routing logic documentation for Enterprise Server 3.3 will match the logic in the Enterprise Cloud version of the article (top right corner at the top of the page). You can see Enterprise Server 3.2 has the old routing logic still and so registration runners are needed to scale from zero with =< 3.2.