Self-hosted runner launching a container fails on checkout

[mpconte]

Describe the bug
File system permission errors occurs on reading

To Reproduce
Using an image built with the following Dockerfile:

FROM ubuntu:20.04

ENV DEBIAN_FRONTEND noninteractive

RUN apt update && apt upgrade -y && apt install ca-certificates -y && apt install wget locales -y && locale-gen en_US.UTF-8     
RUN apt install sudo
RUN apt install gawk wget git-core diffstat unzip texinfo gcc-multilib build-essential chrpath socat libsdl1.2-dev util-linux srecord -y
        
RUN apt install xterm sed cvs subversion coreutils texi2html docbook-utils python-pysqlite2 help2man make gcc g++ desktop-file-utils \
        libgl1-mesa-dev libglu1-mesa-dev mercurial autoconf automake groff curl lzop asciidoc -y

RUN apt install cpio python python3-pip python3-pexpect xz-utils debianutils iputils-ping \
        python3-git python3-jinja2 libegl1-mesa xsltproc fop dblatex xmlto pylint3 -y

RUN apt install u-boot-tools -y

RUN groupadd -r docker && useradd -r -g docker -ms /bin/bash -u 1001 docker && adduser docker sudo && usermod -a -G root docker

After installing the github runner service as a service, launch the following workflow:

name: OS Build

# Controls when the workflow will run

on: 
    push:
      paths-ignore:
        - "Dockerfile"
        - ".github/workflows/docker_build.yml"
        - README.md
    pull_request:
    workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: camis-build-p01
    container:
      image: ghcr.io/sensoftinc/imx8mp_yocto_build_environment_github_runner:1.0.0
      options: -u docker
      credentials:
        username: TraceyThetrace
        password: ${{ secrets.TRACEY_CONTAINER_TOKEN }}

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:      
            
      - uses: actions/checkout@v3

Expected behavior
I expect the checkout step to clone the repo successfully

Runner Version and Platform

Version of your runner? 2.299.1

OS of the machine running the runner? Linux - Ubuntu 20.04

What's not working?

Job Log Output

Run actions/checkout@v3
  with:
    repository: SensoftInc/imx8mp_yocto
    token: ***
    ssh-strict: true
    persist-credentials: true
    clean: true
    fetch-depth: 1
    lfs: false
    submodules: false
    set-safe-directory: true
/usr/bin/docker exec  76bdaee43c8d1f0de18df9364d8d6607c80184b[2](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3525059414/jobs/5911436770#step:3:2)c2dc26a62a2c9516ea6a7[3](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3525059414/jobs/5911436770#step:3:3)ae sh -c "cat /etc/*release | grep ^ID"
node:internal/fs/utils:3[4](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3525059414/jobs/5911436770#step:3:4)4
    throw err;
    ^

Error: EACCES: permission denied, open '/__w/_temp/_runner_file_commands/save_state_9117[5](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3525059414/jobs/5911436770#step:3:5)0dd-b7f8-4c8d-95a5-7a73[6](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3525059414/jobs/5911436770#step:3:6)bec0d9c'
    at Object.openSync (node:fs:585:3)
    at Object.writeFileSync (node:fs:2153:35)
    at Object.appendFileSync (node:fs:2215:6)
    at Object.issueFileCommand (/__w/_actions/actions/checkout/v3/dist/index.js:2293:8)
    at Object.saveState (/__w/_actions/actions/checkout/v3/dist/index.js:118[7](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3525059414/jobs/5911436770#step:3:7)3:31)
    at Object.153 (/__w/_actions/actions/checkout/v3/dist/index.js:4044:10)
    at __webpack_require__ (/__w/_actions/actions/checkout/v3/dist/index.js:22:30)
    at Object.2[8](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3525059414/jobs/5911436770#step:3:8)7 (/__w/_actions/actions/checkout/v3/dist/index.js:7013:34)
    at __webpack_require__ (/__w/_actions/actions/checkout/v3/dist/index.js:22:30)
    at Object.853 (/__w/_actions/actions/checkout/v3/dist/index.js:31801:36) {
  errno: -13,
  syscall: 'open',
  code: 'EACCES',
  path: '/__w/_temp/_runner_file_commands/save_state_[9](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3525059414/jobs/5911436770#step:3:9)[11](https://github.com/SensoftInc/imx8mp_yocto/actions/runs/3525059414/jobs/5911436770#step:3:11)750dd-b7f8-4c8d-95a5-7a736bec0d9c'
}

Runner and Worker's Diagnostic Logs

runner.log

[nikola-jokic]

Hey @mpconte,

Thanks for reporting this. This issue is likely related to the file permissions on a volume mounted to your container.
The container user is running as USER docker with UID 1001. The runner is likely running as a user with UID 1000 or some other UID not matching the 1001. When the runner receives the message, it mounts the volume from the local machine. For simplicity, let's say that your runner user has UID 1000. The directory will be owned by that user. Now, since you provided -u docker as an argument, anything produced from the container will be owned by the UID 1001 on a host machine.
So once you try to checkout, you will have a file permission issue, since user 1001 on a host machine is not allowed to write files to the directory owned by user 1000.

To work around this issue, you should probably provide a user mapping that would match the runner user.

Note: If this does not describe your case, please let me know what user you are running your runner as, so we can try to reproduce this issue and get to the bottom of it ☺️

[nikola-jokic]

I'm going to close out this issue until we hear back from you, please let us know if you are still seeing this issue!

[nuel247]

Run actions/checkout@v3.5.0
with:
repository: camillevilla/diffusion-marketplace
token: ***
ssh-strict: true
persist-credentials: true
clean: true
fetch-depth: 1
lfs: false
submodules: false
set-safe-directory: true
/usr/bin/docker exec 750aa2a7c3ccd6da50f5c16c566eb80f34dcfe2f91f4242b75a390382fe23c79 sh -c "cat /etc/*release | grep ^ID"
node:internal/fs/utils:345
throw err;
^

Error: EACCES: permission denied, open '/__w/_temp/_runner_file_commands/save_state_e8510c3c-b82c-4ab1-8da8-038038b213f9'
at Object.openSync (node:fs:585:3)
at Object.writeFileSync (node:fs:2170:35)
at Object.appendFileSync (node:fs:2232:6)
at Object.issueFileCommand (/__w/_actions/actions/checkout/v3.5.0/dist/index.js:2918:8)
at Object.saveState (/__w/_actions/actions/checkout/v3.5.0/dist/index.js:2835:31)
at Object.8647 (/__w/_actions/actions/checkout/v3.5.0/dist/index.js:2294:10)
at nccwpck_require (/__w/_actions/actions/checkout/v3.5.0/dist/index.js:38410:43)
at Object.2565 (/__w/_actions/actions/checkout/v3.5.0/dist/index.js:146:34)
at nccwpck_require (/__w/_actions/actions/checkout/v3.5.0/dist/index.js:38410:43)
at Object.9210 (/__w/_actions/actions/checkout/v3.5.0/dist/index.js:1099:36) {
errno: -13,
syscall: 'open',
code: 'EACCES',
path: '/__w/_temp/_runner_file_commands/save_state_e8510c3c-b82c-4ab1-8da8-038038b213f9'
}

I am having this issue also, how do i bypass this

[nikola-jokic]

Are you running as a non-root in the container? If yes, the UID from the host machine should match the UID of the user within the container.
This permission issue is usually happening because of that. The checkout tries to write files to the volume mounted directory it does not have permission to write to.

[semihural-tomtom]

@nikola-jokic what is the potential solution here? Should we use 1000 for listenerTemplate?

# listenerTemplate:
#   spec:
#     containers:
#     # Use this section to append additional configuration to the listener container.
#     # If you change the name of the container, the configuration will not be applied to the listener,
#     # and it will be treated as a side-car container.
#     - name: listener
#       securityContext:
#         runAsUser: 1000

[nikola-jokic]

Hey @semihural-tomtom,

Updating listener spec here won't affect the issue, since the listener is not responsible for running the job. The only place where file permissions may cause this problem is on the runner ☺️

[semihural-tomtom]

Hey @nikola-jokic
What is the suggested way of fixing this? I thought maybe changing the securityContext fix this but a bit confused also what will be the suggested/potential solution here

[nikola-jokic]

It is hard for me to reproduce the issue, but when tasked with the file permissions, you can always run the init container that will setup file permissions in the /home/runner/** so you know that the starting point is okay.