Skip to content

Index was out of range when injecting custom containers #3037

Closed as not planned
@mbeliayeu

Description

@mbeliayeu

Checks

Controller Version

0.6.1

Deployment Method

Helm

Checks

  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Use ARC in kubernetes mode

1. Deploy ARC gha-runner-scale-set
2. Configure ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE to use custom pod templates
3. Modify Pod Template to include a sidecar to run alongside `$job` container
or 3. Use Kyverno to inject a sidecar

Describe the bug

When trying to run an action I get an error during the "Initialize containers" step:

  • Error: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index').

The error happens here

Describe the expected behavior

It seems the only way to run sidecars is via service containers. However, It would be nice to have it as a part of Pod spec outside of the workflow definition.

Additional Context

K8S version: 1.28
OS: BottleRocket

Controller's helmfile:

releases:
  - name: arc
    namespace: arc-systems
    chart: arc/gha-runner-scale-set-controller
    version: 0.6.1

Runner's helmfile:

  - name: dev-2gi-runners
    namespace: arc-runners
    chart: arc/gha-runner-scale-set
    version: 0.6.1
    values:
      - githubConfigUrl: ***
        githubConfigSecret: ***
        containerMode:
          type: 'kubernetes'
        template:
          spec:
            securityContext:
              runAsUser: 1001
              runAsGroup: 123
              fsGroup: 123
              seLinuxOptions:
                # See why seLinuxOptions are needed here: https://github.com/bottlerocket-os/bottlerocket/issues/2417
                level: "s0:c123,c456"
            containers:
              - name: runner
                image: ghcr.io/actions/actions-runner:latest
                command: ["/home/runner/run.sh"]
                env:
                  - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE
                    value:  "/home/runner/pod-templates/2gi.yaml"
                  - name: ACTIONS_RUNNER_POD_NAME
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.name
                  - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
                    value: "true"
                  - name: ACTIONS_RUNNER_CONTAINER_HOOKS
                    value: "/home/runner/k8s/index.js"
                volumeMounts:
                  - name: work
                    mountPath: /home/runner/_work
                  - name: pod-templates
                    mountPath: /home/runner/pod-templates
                    readOnly: true
                resources:
                  requests:
                    cpu: 500m
                    memory: 500Mi
            volumes:
              - name: work
                ephemeral:
                  volumeClaimTemplate:
                    spec:
                      accessModes: ["ReadWriteOnce"]
                      storageClassName: "gp3"
                      resources:
                        requests:
                          storage: 30Gi
              - name: pod-templates
                configMap:
                  name: arc-pod-templates
        controllerServiceAccount:
          namespace: arc-systems
          name: arc-gha-rs-controller

pod template (w/o sidecars):

apiVersion: v1
kind: ConfigMap
metadata:
  name: arc-pod-templates
data:
  2gi.yaml: |
    ---
    apiVersion: v1
    kind: PodTemplate
    metadata:
      name: runner-pod-template
      namespace: arc-runners
    spec:
      serviceAccountName: gha-runners-custom
      securityContext:
        runAsUser: 1001
        runAsGroup: 123
        seLinuxOptions:
          level: "s0:c123,c456"
      containers:
        - name: $job
          resources:
            requests:
              cpu: 500m
              memory: 2Gi
            limits:
              memory: 2Gi

Controller Logs

INFO EphemeralRunner Waiting for ephemeral runner owned resources to be deleted
INFO EphemeralRunner Cleaning up the runner pod	{"ephemeralrunner": "arc-runners/dev-2gi-dts5m-runner-tqqxn"}
INFO EphemeralRunner Finalizing ephemeral runner	{"ephemeralrunner": "arc-runners/dev-2gi-dts5m-runner-tqqxn"}

Runner Pod Logs

[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.UpdateJobContext(IExecutionContext context, ContainerInfo jobContainer, List`1 serviceContainers, PrepareJobResponse response)
[WORKER 2023-12-12 18:40:38Z ERR StepsRunner] Caught exception from step: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')

Metadata

Metadata

Assignees

No one assigned

    Labels

    StalebugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions