feat: RunnerSet backed by StatefulSet

[mumoshu]

TL;DR; RunnerSet is a more feature-rich, flexible, easy to configure, and maintainable alternative to RunnerDeployment.

• It can be said feature-rich as it supports Add support for volumeClaimTemplates #612 for using persistent volumes for caching.
• It is flexible as it supports all the pod template settings from StatefulSet API while supporting all the runner-related settings from Runner API
• It is easy to configure and maintainable as the pod-related and container-related settings are now inherited from StatefulSet/Pod Template API. We don't need to maintain and rely on our own variants included in Runner Spec.

---

A runnerset can manage a set of "stateful" runners by combining a statefulset and an admission webhook. A statefulset is a standard Kubernetes construct that manages a set of pods and a pool of persistent volumes. We use that to manage runner pods, while using the admission webhook mutates each pod to have required environment variables and registration tokens.

It is considered to be a complete replacement to the former method of deploying a set of runners, RunnerDeployment, which also creates pods with the required environment variables and registration tokens.

Differences between RunnerSet and RunnerDeployment

The only and big functional difference between RunnerSet and RunnerDeployment is that the former has support for volumeClaimTemplates, which allows actions-runner-controller to manage a pool of dynamically provisioned persistent volumes. This should be useful to make certain types of actions workflows faster by utilizing per-pod-identity cache, like docker layer caches in /var/lib/docker persistent across pod restarts.

The basic usage of RunnerSet is very similar to that of RunnerDeployment.

This RunnerDeployment:

# runnerdeployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: example-runnerdeploy
spec:
  replicas: 2
  template:
    spec:
      repository: mumoshu/actions-runner-controller-ci
      env: []

can be rewritten to:

# runnerset.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: example
spec:
  # NOTE: RunnerSet supports non-ephemeral runners only today
  ephemeral: false
  replicas: 2
  repository: mumoshu/actions-runner-controller-ci
  # Other mandatory fields from StatefulSet
  selector:
    matchLabels:
      app: example
  serviceName: example
  template:
    metadata:
      labels:
        app: example

Also note that, unlike RunnerDeployment, you can write the full StatefulSet spec inside RunnerSet. Configure the pod template however you like, and the runnerset controller reads and tweaks the pod template to create a complete runner pod spec. This makes it unnecessary to add every pod spec fields to runner spec.

How to configure your RunnerSet

You might have written a RunnerDeployment like the below with various tweaks:

# runnerdeployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: example-runnerdeploy
spec:
  replicas: 2
  # Mandatory fields from StatefulSet
  selector:
    matchLabels:
      app: example
  serviceName: example
  # Pod template
  template:
    metadata:
      # Mandatory fields from StatefulSet
      labels:
        app: example
    # Pod template spec
    spec:
      repository: mumoshu/actions-runner-controller-ci
      dockerdWithinRunnerContainer: true
      env: []
      securityContext:
        #All level/role/type/user values will vary based on your SELinux policies.
        #See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/container_security_guide/docker_selinux_security_policy for information about SELinux with containers
        seLinuxOptions: 
          level: "s0"
          role: "system_r"
          type: "super_t"
          user: "system_u"
      resources:
        limits:
          cpu: "4.0"
          memory: "8Gi"
        requests:
          cpu: "2.0"
          memory: "4Gi"
      dockerdContainerResources:
        limits:
          cpu: "4.0"
          memory: "8Gi"
        requests:
          cpu: "2.0"
          memory: "4Gi"

In RunnerDeployment API, you have 4 things to declare in 2 places. 1 thing under spec and 3 things under spec.template.spec:

1. Per-deployment settings like replicas under spec
2. Per-deployment and runner-related settings like repository, organization, enterprise, dockerdWithinRunnerContainer, and so on under spec.template.spec
3. Per-pod settings like securityContext, volumes under spec.template.spec
4. Per-container settings like resources, dockerdContainerResources, image, dockerImage, and so on under spec.template.spec

In RunnerSet API, you have 3 things to declare in 3 places:

1. Per-set settings like replicas, repository, organization, enterprise, and so on under spec
2. Per-pod settings under spec.template.spec
3. Per-container settings under spec.template.spec.containers[]
   • All the dockerdContainer* settings in RunnerDeployment goes to the containers entry whose name is docker, for example.

2 and 3 might be more familiar to many users and therefore it will be easy to write, as it's a standard pod template syntax used widely in Kubernetes Deployment, ReplicaSet, and StatefulSet.

That being said, the above example can be rewritten in RunnerSet like the following:

# runnerset.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: example
spec:
  # NOTE: RunnerSet supports non-ephemeral runners only today
  ephemeral: false
  replicas: 2
  repository: mumoshu/actions-runner-controller-ci
  dockerdWithinRunnerContainer: true
  template:
    spec:
      securityContext:
        #All level/role/type/user values will vary based on your SELinux policies.
        #See https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/container_security_guide/docker_selinux_security_policy for information about SELinux with containers
        seLinuxOptions: 
          level: "s0"
          role: "system_r"
          type: "super_t"
          user: "system_u"
      containers:
      - name: runner
        env: []
        resources:
          limits:
            cpu: "4.0"
            memory: "8Gi"
          requests:
            cpu: "2.0"
            memory: "4Gi"
      - name: docker
        resources:
          limits:
            cpu: "4.0"
            memory: "8Gi"
          requests:
            cpu: "2.0"
            memory: "4Gi"

Planned but not yet implemented

The following features are planned but not implemented. Please use RunnerDeployment for now if you need any of them.

HRA support:
The support for HorizontalRunnerAutoscaler is planned but not done yet.

Scale-from/to-zero:
Scale-from/to-zero is planned but not implemented yet.

Auto-recovery runner pods stuck while registering:
Planned but not implemented yet.

Call for help

I've already verified this to work manually using the updated helm chart and my own build of the actions-runner-controller container image. But, as a lot of changes are made to the code-base, I don't think this is tested enough.

If you want this feature to get merged at all, or get merged earlier, please test and report any problems you encounter!

Changelog

• Add API types, CRDs, controllers/runnerset_contorller.go and controllers/pod_runner_token_injector.go

• Added acceptance/testdata/runnerset.yaml and updated acceptance/deploy.sh. Run a deployment with USE_RUNNERSET=1 to deploy test runners using RunnerSet instead of RunnerDeployment.

• Manually update manager_role.yaml chart template to reflect changes in kustomize config/rbac/role.yaml

• Update controller-gen from 0.3.0 to 0.4.1 to avoid errors generating RunnerSet CRD. The side-effect of this is that all the other CRDs upgraded to v1 API

• Update controller-gen to 0.6.0 to avoid issues on runnerset.spec.template.metaata being empty

• Manually update webhook_configs.yaml char template to reflect changes in kustomize config/webhook/manifests.yaml

• Upgraded controller-runtime to 0.9.0 to prevent envtest failing to install v1 CRDs

  • I had to fix various errors due to deprecation and removals listed in e.g. https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.7.0
    Issues I have encountered while developing this:

• Default value not allowed for in x-kubernetes-list-map-keys kubernetes-sigs/controller-tools#444 (comment)

• Single-version generated CRDs cannot be installed kubernetes-sigs/controller-tools#302

Related issues

Resolves #613
Ref #612
Revival of #4

[callum-tait-pbx]

if all the images used in acceptance/load aren't already on your local machine then this fails. acceptance/pull needs to be done after acceptance/kind

[callum-tait-pbx]

Worth bumping to v1.1.1 in this PR?

v1.1.1 is the last of the v1.X.X series. I run v1.1.1 on EKS and have done across multiple controller versions. v1.1.1 is fairly old at this point but we should consider bumping it but a newer major version outside of this PR so if there are issues (I don't see why there would be tbh) they are dealt with seperately to this work. I can vouch for v1.1.1 so it would be nice to bump to latest of that series in this PR seen as we have done various bumps already.

Perhaps it's worth having CERT_MANAGER_VERSION = v1.1.1 at the top and the version to be deployed pulled from that making it easier to bump next time?

[esvirskiy]

Hi @mumoshu. I am testing the controller using the canary tag and I am seeing the following error

actions-runner-controller-67bc455dd6-css9q manager E0622 15:10:58.821753       1 leaderelection.go:325] error retrieving resource lock actions-runner-system/actions-runner-controller: leases.coordination.k8s.io "actions-runner-controller" is forbidden: User "system:serviceaccount:actions-runner-system:actions-runner-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "actions-runner-system"

I see that https://github.com/actions-runner-controller/actions-runner-controller/blob/8b90b0f0e3a4a254c096f8d9ecd8aeed0ee3c00e/controllers/runnerset_controller.go#L68 is commented out. Is that needed?

[esvirskiy]

Hi @mumoshu. I am testing the controller using the canary tag and I am seeing the following error

actions-runner-controller-67bc455dd6-css9q manager E0622 15:10:58.821753       1 leaderelection.go:325] error retrieving resource lock actions-runner-system/actions-runner-controller: leases.coordination.k8s.io "actions-runner-controller" is forbidden: User "system:serviceaccount:actions-runner-system:actions-runner-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "actions-runner-system"

I see that

https://github.com/actions-runner-controller/actions-runner-controller/blob/8b90b0f0e3a4a254c096f8d9ecd8aeed0ee3c00e/controllers/runnerset_controller.go#L68

is commented out. Is that needed?

This was my fault. I used an outdated chart (changed controller tag to canary).
The chart that is currently in master works fine. Thanks! I'll continue testing this!

[mumoshu]

@esvirskiy Wow! Thanks a lot for testing ☺️ Please feel free to leave any early feedbacks. That would be super helpful to shape this feature.

Note that there's a few unimplemented things as explained in the pr description:

I'm working for the HRA support at #647. I'll tackle the auto-recovery feature next. Scale-from/to-zero is at the lowest priority and I may skip working on it entirely, because a potential enhancement on the GitHub side can make it unnecessary.