Workflow fails to pull image from private registry due to missing authentication

[diegoara96]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.9.3

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

create a worflow that uses a service from a private repository as follows:

      postgresdb:
        # Docker Hub image
        image: my.private.registry/project/postgresdb:latest
        credentials:
          username: ${{ secrets.HARBOR_USER }}
          password: ${{ secrets.HARBOR_PASS }}

Describe the bug

When creating a workflow that needs to use images hosted in a private repository we see a “no basic auth credentials” error. The same workflow running against a normal self-hosted runner is able to download the image.

I opened this discussion #3741 thinking it was a question but seeing that with the normal self-runner it works I understand that it is a bug.

Describe the expected behavior

It should work just like a self-hosted runner and be able to download the image from the private registry.

Additional Context

githubConfigUrl: "URL"

githubConfigSecret:
  ### GitHub PAT Configuration
  github_token: "ghp_TOKEN"

## maxRunners is the max number of runners the autoscaling runner set will scale up to.
#maxRunners: 5

## minRunners is the min number of idle runners. The target number of runners created will be
## calculated as a sum of minRunners and the number of jobs assigned to the scale set.
minRunners: 1

containerMode:
  type: "kubernetes"  ## type can be set to dind or kubernetes
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "rook-cephfs"
    resources:
      requests:
        storage: 2Gi

  spec:
    securityContext:
      fsGroup: 1001
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:latest
      command: ["/home/runner/run.sh"]
      env:
 #          - name: ACTIONS_RUNNER_CONTAINER_HOOKS
 #            value: /home/runner/k8s/index.js
 #          - name: ACTIONS_RUNNER_POD_NAME
 #            valueFrom:
 #              fieldRef:
 #                fieldPath: metadata.name
        - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
          value: "false"
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
      resources:
        requests:
          memory: "1Gi"
          cpu: "2"
        limits:
          memory: "4Gi"
          cpu: "4"
          
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: [ "ReadWriteOnce" ]
              storageClassName: "rook-cephfs"
              resources:
                requests:
                  storage: 2Gi

Controller Logs

https://gist.github.com/diegoara96/20c6ce4fd17ef006de796bb1266b196d

Runner Pod Logs

workflow describe: https://gist.github.com/diegoara96/e75fa18237a7643ecb71d34cce562bc9
runner log: https://gist.github.com/diegoara96/47a20be90c587594bf25605053642524

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[diegoara96]

This functionality was added by github in 2020 (changelog) but from what I see it doesn't seem to have ever been implemented here. It seems that the workflow credentials object is not used by te ARC

[chrispat]

Which mode are you using ARC in for container jobs? Docker-in-docker or Kubernetes?

[diegoara96]

Hi @chrispat
We are using kubernetes mode. You have all the configuration in Additional Context in the first message.

[diegoara96]

Hi @chrispat were you able to take a look at this problem?

[diegoara96]

@TingluoHuang @Link- Any idea what might be going on?

[ArieLevs]

Hi we are facing similar issue, both when using kubernetes or dind mode.
not only we cannot provide credentials using Defining credentials for a container registry

we use EKS, so we don't even have a static token we can provide, each time a job runs it gets a short lived token to reach AWS resources (in my case ECR)

solving this issue on k8s level is pretty straightforward (image pull secrets/serviceaccount IRSA/etc)

what do we do in order to provide credential to this internal process? (running aws ecr get-login-password ....? where?)

[gitandre]

And this related to intermitent docker sock errors - not for this PR but still an issue as we are getting the same errors

https://github.com/actions/actions-runner-controller/discussions/2999

actions/actions-runner-controller#3842

[ArieLevs]

Hi we are facing similar issue, both when using kubernetes or dind mode. not only we cannot provide credentials using Defining credentials for a container registry

we use EKS, so we don't even have a static token we can provide, each time a job runs it gets a short lived token to reach AWS resources (in my case ECR)

solving this issue on k8s level is pretty straightforward (image pull secrets/serviceaccount IRSA/etc)

what do we do in order to provide credential to this internal process? (running aws ecr get-login-password ....? where?)

If anyone's interested I've solved this issue using hook extensions.
Implementing a sidecar container (building images using Podman that needs to pull/pull from ecr) that executes aws get-login ... and shares the token via empty dir, it works but feels like a complex implementioan, would love to see GitHub arc to be more Kubernetes native