Index was out of range when injecting custom containers

[mbeliayeu]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.6.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Use ARC in kubernetes mode

1. Deploy ARC gha-runner-scale-set
2. Configure ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE to use custom pod templates
3. Modify Pod Template to include a sidecar to run alongside `$job` container
or 3. Use Kyverno to inject a sidecar

Describe the bug

When trying to run an action I get an error during the "Initialize containers" step:

• Error: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index').

The error happens here

Describe the expected behavior

It seems the only way to run sidecars is via service containers. However, It would be nice to have it as a part of Pod spec outside of the workflow definition.

Additional Context

K8S version: 1.28
OS: BottleRocket

Controller's helmfile:

releases:
  - name: arc
    namespace: arc-systems
    chart: arc/gha-runner-scale-set-controller
    version: 0.6.1

Runner's helmfile:

  - name: dev-2gi-runners
    namespace: arc-runners
    chart: arc/gha-runner-scale-set
    version: 0.6.1
    values:
      - githubConfigUrl: ***
        githubConfigSecret: ***
        containerMode:
          type: 'kubernetes'
        template:
          spec:
            securityContext:
              runAsUser: 1001
              runAsGroup: 123
              fsGroup: 123
              seLinuxOptions:
                # See why seLinuxOptions are needed here: https://github.com/bottlerocket-os/bottlerocket/issues/2417
                level: "s0:c123,c456"
            containers:
              - name: runner
                image: ghcr.io/actions/actions-runner:latest
                command: ["/home/runner/run.sh"]
                env:
                  - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE
                    value:  "/home/runner/pod-templates/2gi.yaml"
                  - name: ACTIONS_RUNNER_POD_NAME
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.name
                  - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
                    value: "true"
                  - name: ACTIONS_RUNNER_CONTAINER_HOOKS
                    value: "/home/runner/k8s/index.js"
                volumeMounts:
                  - name: work
                    mountPath: /home/runner/_work
                  - name: pod-templates
                    mountPath: /home/runner/pod-templates
                    readOnly: true
                resources:
                  requests:
                    cpu: 500m
                    memory: 500Mi
            volumes:
              - name: work
                ephemeral:
                  volumeClaimTemplate:
                    spec:
                      accessModes: ["ReadWriteOnce"]
                      storageClassName: "gp3"
                      resources:
                        requests:
                          storage: 30Gi
              - name: pod-templates
                configMap:
                  name: arc-pod-templates
        controllerServiceAccount:
          namespace: arc-systems
          name: arc-gha-rs-controller

pod template (w/o sidecars):

apiVersion: v1
kind: ConfigMap
metadata:
  name: arc-pod-templates
data:
  2gi.yaml: |
    ---
    apiVersion: v1
    kind: PodTemplate
    metadata:
      name: runner-pod-template
      namespace: arc-runners
    spec:
      serviceAccountName: gha-runners-custom
      securityContext:
        runAsUser: 1001
        runAsGroup: 123
        seLinuxOptions:
          level: "s0:c123,c456"
      containers:
        - name: $job
          resources:
            requests:
              cpu: 500m
              memory: 2Gi
            limits:
              memory: 2Gi

Controller Logs

INFO EphemeralRunner Waiting for ephemeral runner owned resources to be deleted
INFO EphemeralRunner Cleaning up the runner pod	{"ephemeralrunner": "arc-runners/dev-2gi-dts5m-runner-tqqxn"}
INFO EphemeralRunner Finalizing ephemeral runner	{"ephemeralrunner": "arc-runners/dev-2gi-dts5m-runner-tqqxn"}

Runner Pod Logs

[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.UpdateJobContext(IExecutionContext context, ContainerInfo jobContainer, List`1 serviceContainers, PrepareJobResponse response)
[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] Caught exception from step: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[wd-hopkins]

We are facing the same issue

[mbeliayeu]

We are facing the same issue

I ended up adding a sidecar explicitly in the GitHub action (under jobs.<job_id>.services), and configuring it via Kyverno.

[github-actions]

This issue is stale because it has been open 365 days with no activity. Remove stale label or comment or this will be closed in 15 days.

[github-actions]

This issue was closed because it has been stalled for 15 days with no activity.