Support Runner inside of Docker Container

[jpb]

Describe the enhancement

Fully support all features when runner is within a Docker container.

Not all features are currently supported when the runner is within a Docker container, specifically those features that use Docker like Docker-based Actions and services. Running self-hosted runners using Docker is an easy way to scale out runners on some sort of Docker-based cluster and an easy way to provide clean workspaces for each run (with ./run.sh --once).

Code Snippet

Possible implementation that I am using now.

Additional information

There are a few areas of concern when the runner executes in a Docker container:

1. Filesystem access for other containers needed as part of the job. This can be resolved by using a volume mount from the host which uses a matching host and container path (for example: docker run -v /home/github:/home/github, although it doesn't have to be this exact directory) and telling the runner to use a directory within that for the work directory (./config.sh --work /home/github/work). This works with the current volume mounting behaviour for containers created by the runner. This would need to be documented as part of the setup process for a Docker-based runner.
2. Network between runner and other containers needed as part of the job. This could be resolved by not creating a network as part of the run and instead optionally accepting an existing network to be used. I have found that it works well to use --network container:<container ID of the runner> to reuse the network from the runner container without having to orchestrate a network created via docker network create. There is no straightforward way to discover the network or ID of a container from within it, so it would likely need to be the responsibility of the user to pass this information to the runner (I current do something like "container:$(cat /proc/self/cgroup | grep "cpu" | head -n 1 | rev | cut -d/ -f 1 | rev)" from within the runner container to find the ID and pass this to the runner, although this isn't guaranteed to work in all cases).

[jpb]

There appear to be a couple more things that need to be done to account for multiple runners on the same host concurrently:

1. docker network prune can not run concurrently and should likely be retried if such an error is received:

   /usr/local/bin/docker network prune --force --filter "label=898d1dec6adc"
   Error response from daemon: a prune operation is already running
   ##[warning]Delete stale container networks failed, docker network prune fail with exit code 1
   

2. The docker label is not sufficient for isolating separate runners on the same host. The current hash of the root directory will result in the same label being used for all runners with the exact same version. In my testing I've switched this to use the hostname, but perhaps something like the runner name or run ID could be used.

[jpb]

@TingluoHuang @bryanmacfarlane I'm hoping to get your feedback on this – getting official support for this would be a huge help for me. I'm happy to work on an implementation if that is helpful.

[SonicGD]

This is a big problem for us. We want to run gh runner in docker to easier scaling and isolation. But we need to also run services for tests. So our workaround for now is to run multiple runners on host, but scaling container with docker-compose is so much easier and convenient.

[npalm]

This is a big problem for us. We want to run gh runner in docker to easier scaling and isolation. But we need to also run services for tests. So our workaround for now is to run multiple runners on host, but scaling container with docker-compose is so much easier and convenient.

Fully agree, for the time being we have build a scalable solution on AWS spot to server our docker builds. A detailed blog post en ref to the code https://040code.github.io/2020/05/25/scaling-selfhosted-action-runners

[jupe]

This is a big problem for us. We want to run gh runner in docker to easier scaling and isolation. But we need to also run services for tests. So our workaround for now is to run multiple runners on host, but scaling container with docker-compose is so much easier and convenient.

Currently we have to create workarounds using non-optimal solutions to deploy tens of runners - or workarounds for container usage in jobs which is rather ugly. How to raise priority of this ?

Just curious, how other's manage scaling the runners ? This is probably one of the most interested approach so far I've seen.. I guess many of us faced this same challenge when scaling gh-runners.. "Official" scaling proposals from GitHub would be more than welcome :) .

[vincentbrison]

This is a big problem for us. We want to run gh runner in docker to easier scaling and isolation. But we need to also run services for tests. So our workaround for now is to run multiple runners on host, but scaling container with docker-compose is so much easier and convenient.

Big kudos for @npalm and its solution on AWS. We also build a similar solution for GCP allowing us to scale our self hosted runners for a whole GitHub organization https://github.com/faberNovel/terraform-gcp-github-runner