Unable to run container jobs using services.github-runners

[eliandoran]

Describe the bug

Jobs fail when using a container: declaration in GitHub workflows, using a non-NixOS image.

The container is well built and run, but the all the GitHub Actions run inside it will crash with:

/nix/store/jk6qarf9kzfl49v7mgpyhwmg1znhcax0-docker-24.0.5/bin/docker exec  6b346de7d0c9f8d99cdca256f1119c5d2af04625baa91e3ad1d0f4afbf978953 sh -c "cat /etc/*release | grep ^ID"
OCI runtime exec failed: exec failed: unable to start container process: exec: "/__e/node20/bin/node": stat /__e/node20/bin/node: no such file or directory: unknown

Adding a rudimentary log step reveals that the node executable is there:

ls -l /__e
total 0
lrwxrwxrwx 1 root root 57 Jan  1  1970 node20 -> /nix/store/6wq28hcbiclvmwgphry7jsy3w7mzypqc-nodejs-20.9.0

Steps To Reproduce

Steps to reproduce the behavior:

1. Set up a GitHub runner using services.github-runners.
2. Define a GitHub action like this one:

jobs:
  build:
    container:
      image: ubuntu:latest
    steps:
      - name: Checkout the repository
        uses: actions/checkout@v4

Expected behavior

GitHub Actions should execute OK even when run under containers.

Additional context

My suspicion is that we have a non-NixOS container (e.g. debian) that is trying to run to a NixOS executable (since the path to the symlink above does point to the Nix store).

Notify maintainers

@veehaitch , @thomasjm

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

Will fill it later, since I don't have access to the runner at this moment.

---

Add a 👍 reaction to issues you find important.

[thomasjm]

Your suspicion sounds right. I'm not really sure how this works in a normal container: stanza outside of NixOS. I guess the GitHub Runner bind-mounts the necessary tools into the container?

In this case, it seems like it may be bind-mounting the Node.JS directory, but not the rest of its closure in the Nix store, so it fails to find shared libraries.

Maybe it'll work if you do this?

    container:
      image: ubuntu:latest
      volumes:
        - /nix:/nix

[eliandoran]

@thomasjm , you're absolutely right. Mounting /nix in the container does seem to solve the issue.
Now I'm wondering how we could fix this without having to change the workflow itself (since it could cause compatibility issues when using the workflow on non-NixOS runners).

I think the issue stems from the dynamic library loader, having a hard-coded path to the loader in the Nix store.

[thomasjm]

Probably not just the loader, but potentially all runtime dependencies of the chosen Node.js.

I think the right solution would be to configure GitHub runners on NixOS to mount the runtime closure of Node.js (not the full /nix dir, as that seems like too much) using the instructions here:

https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/customizing-the-containers-used-by-jobs

tl;dr from the docs: it looks like we need to set the ACTIONS_RUNNER_CONTAINER_HOOKS environment variable for the runner to point to an index.js where we define a prepare_job script, and the script should define jobContainer.systemMountVolumes to mount the paths we need.