RunnerScaleSet in dind mode failing with startupProbe: Forbidden: may not be set for init containers without restartPolicy=Always error in controller

[jlehoux-cvet]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.12.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Deploy a AutoscalingRunnerSet using in dind mode with no customization to the template.spec parameters.

Describe the bug

AutoscalingRunnerSet gets created, but no runners pods are created. Controller has the following error in the logs:

ERROR EphemeralRunner Failed to create a pod due to unrecoverable failure {"version": "0.12.0", "ephemeralrunner": {"name":"","namespace":"arc-runners"}, "error": "Pod "" is invalid: spec.initContainers[1].startupProbe: Forbidden: may not be set for init containers without restartPolicy=Always"}

When i output the autoscalingrunnerset yaml on the cluster, it is wrong and has dind setup as an initcontainer:

spec:
  githubConfigSecret: github-auth-cvet-test
  githubConfigUrl: https://github.com/Cvet-test
  maxRunners: 3
  minRunners: 1
  runnerScaleSetName: cvet-test
  template:
    spec:
      containers:
      - command:
        - /home/runner/run.sh
        env:
        - name: DOCKER_HOST
          value: unix:///var/run/docker.sock
        - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
          value: "120"
        image: ghcr.io/actions/actions-runner:latest
        name: runner
        volumeMounts:
        - mountPath: /home/runner/_work
          name: work
        - mountPath: /var/run
          name: dind-sock
      initContainers:
      - args:
        - -r
        - /home/runner/externals/.
        - /home/runner/tmpDir/
        command:
        - cp
        image: ghcr.io/actions/actions-runner:latest
        name: init-dind-externals
        volumeMounts:
        - mountPath: /home/runner/tmpDir
          name: dind-externals
      - args:
        - dockerd
        - --host=unix:///var/run/docker.sock
        - --group=$(DOCKER_GROUP_GID)
        env:
        - name: DOCKER_GROUP_GID
          value: "123"
        image: docker:dind
        name: dind
        restartPolicy: Always
        securityContext:
          privileged: true
        startupProbe:
          exec:
            command:
            - docker
            - info
          failureThreshold: 24
          initialDelaySeconds: 0
          periodSeconds: 5
        volumeMounts:
        - mountPath: /home/runner/_work
          name: work
        - mountPath: /var/run
          name: dind-sock
        - mountPath: /home/runner/externals
          name: dind-externals
      restartPolicy: Never
      serviceAccountName: cvet-test-gha-rs-no-permission
      volumes:
      - emptyDir: {}
        name: dind-sock
      - emptyDir: {}
        name: dind-externals
      - emptyDir: {}
        name: work

Describe the expected behavior

Dind runners are successfully created

Additional Context

values.yaml:

githubConfigUrl: "https://github.com/Cvet-test"
containerMode:
  type: "dind"
minRunners: 1
maxRunners: 3
githubConfigSecret: github-auth-cvet-test
runnerScaleSetName: cvet-test

Controller Logs

https://gist.github.com/jlehoux-cvet/976db1e422825548496c1b8c558ab7a1

Runner Pod Logs

No runner is created, therefore there are no logs.

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[nimjor]

What Kubernetes version is your cluster running?

[jlehoux-cvet]

What Kubernetes version is your cluster running?

1.31.9

[jlehoux-cvet]

In order to attempt to move forward I also removed the containerMode.type setting from my values.yaml and manually specified a template.spec, which did allow me to get further, but with that setup i am having issues where even though i have given the runner pods a service account with an iam role linked, it isnt being honored and the iam role of the ec2 instance is being used, which doesnt have permission to push to ecr.

[poonam2309]

Try running removing the restart policy parameter from the values.yaml Thanks & Regards... Poonam Kadian

…

On Wed, 25 Jun, 2025, 9:44 pm jlehoux-cvet, ***@***.***> wrote: *jlehoux-cvet* left a comment (actions/actions-runner-controller#4144) <#4144 (comment)> In order to attempt to move forward I also removed the containerMode.type setting from my values.yaml and manually specified a template.spec, which did allow me to get further, but with that setup i am having issues where even though i have given the runner pods a service account with an iam role linked, it isnt being honored and the iam role of the ec2 instance is being used, which doesnt have permission to push to ecr. — Reply to this email directly, view it on GitHub <#4144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQCC7N5FGSBAF665GLSPYQT3FLDETAVCNFSM6AAAAACABLSOPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMBVGM2TSOJXGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[nimjor]

it is wrong and has dind setup as an initcontainer

This is not wrong, it is an intended new feature: #3842

An initContainer with restartPolicy = Always is known as a sidecar container:

Sidecar containers are the secondary containers that run along with the main application container within the same Pod. These containers are used to enhance or to extend the functionality of the primary app container by providing additional services, or functionality such as logging, monitoring, security, or data synchronization, without directly altering the primary application code.

This exactly describes the purpose of the dind container. Running it as a sidecar (initContainer with restartPolicy = Always) ensures it is started before the runner and remains running for the entire lifecycle of the runner container, which is important since the docker client in the runner relies on dind to provide the docker daemon.

The SidecarContainers feature has been enabled by default since k8s 1.29 (as a beta feature) but is only general availability since 1.33. Does your cluster have the feature enabled?

[poonam2309]

Try this configuration

  githubConfigSecret: github-auth-cvet-test
  githubConfigUrl: https://github.com/Cvet-test
  maxRunners: 3
  minRunners: 1
  runnerScaleSetName: cvet-test
  template:
    spec:
      initContainers:
      - args:
        - -r
        - /home/runner/externals/.
        - /home/runner/tmpDir/
        command:
        - cp
        image: ghcr.io/actions/actions-runner:latest
        name: init-dind-externals
        volumeMounts:
        - mountPath: /home/runner/tmpDir
          name: dind-externals
     containers:
      - command:
        - /home/runner/run.sh
        env:
        - name: DOCKER_HOST
          value: unix:///var/run/docker.sock
        - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
          value: "120"
        image: ghcr.io/actions/actions-runner:latest
        name: runner
        volumeMounts:
        - mountPath: /home/runner/_work
          name: work
        - mountPath: /var/run
          name: dind-sock
     - name: dind
        args:
          - dockerd
          - --host=unix:///var/run/docker.sock
          - --group=$(DOCKER_GROUP_GID)
        env:
            - name: DOCKER_GROUP_GID
               value: "123"
        image: docker:dind
        securityContext:
          privileged: true
        startupProbe:
          exec:
            command:
            - docker
            - info
          failureThreshold: 24
          initialDelaySeconds: 0
          periodSeconds: 5
        volumeMounts:
        - mountPath: /home/runner/_work
          name: work
        - mountPath: /var/run
          name: dind-sock
        - mountPath: /home/runner/externals
          name: dind-externals
    serviceAccountName: cvet-test-gha-rs-no-permission
    volumes:
     - emptyDir: {}
        name: dind-sock
     - emptyDir: {}
        name: dind-externals
     - emptyDir: {}
        name: work

[jlehoux-cvet]

@poonam2309 I have attempted that, and I get past the forbidden error, but even though i have given the runner pods a serviceAccount with an associated IAM role, it is not being honored, even though the AWS env vars are correctly configured on the containers, the workflow still tries to use the IAM role of the ec2 instance.

[jlehoux-cvet]

The SidecarContainers feature has been enabled by default since k8s 1.29 (as a beta feature) but is only general availability since 1.33. Does your cluster have the feature enabled?

Yes
kubernetes_feature_enabled{name="SidecarContainers",stage="BETA"} 1

[jlehoux-cvet]

I am running out of time for this project and will likely need to revert to the summerwinds setup which works without any of these headaches. I will attempt to keep working on this on the side, but responses may become slower.

[nikola-jokic]

Hey @jlehoux-cvet,

You should leave the container mode empty. It is noted in the values.yaml file here, if you are modifying/customising the spec for container mode, you should leave the containerMode empty, and apply your spec.

As for the service account, I'm not sure how you are applying it. The chart doesn't have any special handling for it, so please debug it using --dry-run. If you feel like we are mishandling the service account field, please submit another issue.

I'm going to close this issue here since the original issue doesn't apply if you follow the guide from the base values.yaml file.