Experiencing `permission denied` after v3.1.0 on jobContainers when container user is not 'root'

[fhammerl]

After the rollout actions/checkout v3.1.0, some reported permission denied errors when using the action on a jobContainer running with a container user that is not 'root'. With v3.1.0, we're using commands that write to files to replace deprecated stdout commands.

These are the files in question causing permission issues due to mismatched UIDs between user running the runner process and the user on the container it starts.

Running a container using a container user that is not 'root' is an unsupported path, we've listed some workarounds below.

Summary

In short: the jobContainers in these workflows seem to follow an unsupported path. Namely,

Do not use the USER instruction in your Dockerfile, because you won't be able to access the GITHUB_WORKSPACE

I believe the jobContainer operates as the user 'circleci' in the workflow you linked.

We will roll out the changes again as we haven't seen issues with the officially supported paths.

It is recommended 'not use the USER instruction in your Dockerfile'.
In most cases, this means 'run the container as root'.

I understand this introduces some friction, so see below for workarounds and a technical summary.

Detailed Summary

1. In actions/checkout@3.1.0, we began phasing out the use of some workflow commands that write to stdout with functions that write to temp files instead.
2. If you're running in a jobContainer, these temp files are created by the runner app on your host OS and are owned by the user who started the runner (in case of ubuntu-latest, this user is 'runner' with UID-1001). These files are then mounted to your jobContainer.
3. actions/checkout is a .js file executed using node on your jobContainer. As per the changes in 1., actions/checkout@v3.1.0 attempts to write to these temp files on your jobContainer.

This can have a couple of outcomes:

1. If the jobContainer user is 'root': writes successfully.
2. If the jobContainer user is NOT 'root':
   a) If the UID of the jobContainer user == UID of host runner user (UID-1001 on ubuntu-latest): write is successful
   b) If the UID of the jobContainer user != UID of host runner user (UID-1001 on ubuntu-latest): permission denied

The jobContainers in the above workflow seem to run into 2/b.

Fix

As per the docs, don't modify the default user in the Dockerfile. If that's not feasible for you: 👇

Workarounds (unsupported)

• Use actions/checkout@v3.0.2 in your workflow (will be deprecated)
• Match the UID of the host runner user (e.g. UID-1001 on ubuntu-latest today) to the UID of the user on the jobContainer
• Override the default container user and use 'root':

container: 
    image: alpine:latest
    options: --user root

Error logs

##[debug]Running JavaScript Action with default external tool: node16
node:internal/fs/utils:344
    throw err;
    ^

Error: EACCES: permission denied, open '/__w/_temp/_runner_file_commands/save_state_2[7]

[rGaillard]

We experience the same issue with the v2.5.0 version, our CI is broken, had to rollback :(

[priv-kweihmann]

The same happens with latest release on actions/cache - the common denominator seems to the usage of actions/core 1.10.0.
All previous versions that do not use this particular core version are fine

[chenrui333]

Anyone trying to fix this?

[Bo98]

Just brainstorming a bit here: would there be security implications if the runner were to set o+w for file commands? I guess it maybe could for some specific multi-user self-hosted scenarios, but for GitHub-hosted runners and self-hosted runners that opt-in/out: it could make sense and seems like something that could ease concerns here a bit.

Match the UID of the host runner user (e.g. UID-1001 on ubuntu-latest today) to the UID of the user on the jobContainer

I guess the core problem is Docker doesn't really support UID remaps at runtime itself.

Nevertheless, this is a solution I was initially going to chase but the problem is that any mechanism to do so dynamically is disabled in GitHub Actions. While you could rely on a fixed 1001 for GitHub-hosted, it gets trickier with varying self-hosted setups. Initial idea I had was to create an entrypoint that would get the UID of RUNNER_WORKSPACE and use that to change the container user, but unfortuately entrypoints are force disabled unless container hooks are enabled (not entirely sure why), which obviously rules out it working on GitHub-hosted.