Error: Error: Missing file at path: /home/runner/_work/_temp/_runner_hook_responses/{...}.json

[jennifgcrl]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.11.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Start with an EKS auto cluster
2. Follow quick start instructions https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/quickstart-for-actions-runner-controller
3. Follow the kubernetes runner instructions https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#using-docker-in-docker-or-kubernetes-mode-for-containers
4. Follow this troubleshooting step https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors#error-access-to-the-path-homerunner_work_tool-is-denied
5. Launch any github action that runs on a container

Describe the bug

The github action fails on the step "Initializing containers" with the message:

Run '/home/runner/k8s/index.js'
  shell: /home/runner/externals/node20/bin/node {0}
Error: Error: Missing file at path: /home/runner/_work/_temp/_runner_hook_responses/{...}.json
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.

The workflow runner (the -workflow pod) starts and instantly dies, printing no logs.

I've also tried fsGroup: 1001 and it didn't work either. Without fsGroup I get the /home/runner/_work/_tool access error.

Describe the expected behavior

The action should progress past "Initializing containers". The workflow pod presumably should print something.

Additional Context

githubConfigUrl: "https://github.com/{...}/{...}"
githubConfigSecret: {...}
maxRunners: 2
containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "auto-ebs-sc"
    resources:
      requests:
        storage: 4Gi
template:
  spec:
    securityContext:
      fsGroup: 123

Controller Logs

https://gist.github.com/jennifgcrl/0d17f852acf4ea954d631cd10bfb90ae

I don't think there's anything interesting here...

Runner Pod Logs

(runner pod logs empty)

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[nikola-jokic]

Hey @jennifgcrl,

If you don't use the fs group, does it work? Please submit the runner log. This issue doesn't seem to be related to ARC, but let's keep it open here, and maybe we can help troubleshoot the issue.

[jennifgcrl]

Without the fs group it shows "Error: Access to the path /home/runner/_work/_tool is denied"

Here's the runner log: https://gist.github.com/jennifgcrl/e6b770387e248e7bbe59b3b9fa16fdda

The relevant lines seem to be:

arc-runner-set-wtqrv-runner-l7w65 runner [WORKER 2025-04-03 13:13:31Z ERR  StepsRunner]  ---> System.UnauthorizedAccessException: Access to the path '/home/runner/_work/_temp/_runner_hook_responses' is denied.
arc-runner-set-wtqrv-runner-l7w65 runner [WORKER 2025-04-03 13:13:31Z ERR  StepsRunner]  ---> System.IO.IOException: Permission denied

[nikola-jokic]

What image are you using for the runner?

[jennifgcrl]

Runner is ghcr.io/actions/actions-runner:latest

Workflow is:

name: Hello containers
run-name: Hello, containers!
on:
  push:
    branches:
      - master
jobs:
  build-builder:
    runs-on: arc-runner-set
    container:
      image: ubuntu:latest
    steps:
      - run: echo "Hello, world!"

[xmath279]

I have the same issue. I do not have anything to add, as my setup is pretty much identical to @jennifgcrl.

I wonder @nikola-jokic, how can this not be an ARC issue? It occurs when running the simplest workflow and with a simple ARC installation.

@jennifgcrl: have you been able to fix this issue?

[jennifgcrl]

No, I worked around it by not using the k8s containermode. I rewrote my actions to run in the default arc image and launch the actual job in a separate pod.

[nikola-jokic]

Hey @jennifgcrl,

I failed to reproduce this error. Can you please clean up the entire cluster and start from scratch? Then, please provide the full controller log. I noticed the controller had errors previously, so it might be interfering with the k8s mode configuration.

[xmath279]

Hello @nikola-jokic, I was able to get this fixed in the end by adding a securityContext to the runner pod. The issue seems to be due to default security settings of the Bottlerocket AMI.

      securityContext:
        seLinuxOptions:
          level: "s0"
          role: "system_r"
          type: "super_t"
          user: "system_u"

I have found the fix in this issue: actions/runner#981 (comment)

You can probably close this issue if @jennifgcrl also confirms it fixes it for her, but it may be helpful to add this to an FAQ or something, as I had to look for the solution for quite some time.

[jennifgcrl]

Amazing! It fixed it for me too. Thanks!

[nikola-jokic]

Thank you both for confirming!