Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rootless DinD runner #1644

Merged
merged 3 commits into from
Aug 3, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1163,6 +1163,10 @@ spec:
env: []
```

#### Runner with rootless DinD

When using the DinD runner, it assumes that the main runner is rootful, which can be problematic in a regulated or more security-conscious environment, such as co-tenanting across enterprise projects. The `actions-runner-dind-rootless` image runs rootless Docker inside the container as `runner` user. Note that this user does not have sudo access, so anything requiring admin privileges must be built into the runner's base image (like running `apt` to install additional software).

#### Runner with K8s Jobs

When using the default runner, jobs that use a container will run in docker. This necessitates privileged mode, either on the runner pod or the sidecar container
Expand Down
139 changes: 139 additions & 0 deletions runner/actions-runner-dind-rootless.dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
FROM ubuntu:20.04

# Target architecture
ARG TARGETPLATFORM=linux/amd64

# GitHub runner arguments
ARG RUNNER_VERSION=2.294.0

# Docker and Docker Compose arguments
ENV CHANNEL=stable
ARG COMPOSE_VERSION=v2.6.0

# Dumb-init version
ARG DUMB_INIT_VERSION=1.2.5

# Other arguments
ARG DEBUG=false

# Set environment variables needed at build
ENV DEBIAN_FRONTEND=noninteractive

RUN apt update -y \
&& apt-get install -y software-properties-common \
&& add-apt-repository -y ppa:git-core/ppa \
&& apt-get update -y \
&& apt-get install -y --no-install-recommends \
build-essential \
curl \
ca-certificates \
dnsutils \
ftp \
fuse-overlayfs \
Copy link
Contributor

@isarkis isarkis Jul 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had this PR tested on our self hosted runners infra in GKE. We found fuse-overlayfs not working very well on Ubuntu (super slow, deadlocking, etc.). Everything is working as expected when using default overlay filesystem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isarkis, I think it's a bit out of scope for this PR - I superimposed my work on this topic from my demo repository onto this one for the PR. I don't run fuse-overlayfs on my own images either, for much the same reason.

Copy link
Contributor

@isarkis isarkis Jul 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then, let's remove fuse-overlayfs from the runner/actions-runner-dind-rootless.dockerfile. Per https://docs.docker.com/engine/security/rootless/, overlay2, which is the default, is the recommended file system for Ubuntu.

git \
iproute2 \
iputils-ping \
iptables \
jq \
libunwind8 \
locales \
netcat \
net-tools \
openssh-client \
parallel \
python3-pip \
rsync \
shellcheck \
supervisor \
software-properties-common \
sudo \
telnet \
time \
tzdata \
uidmap \
unzip \
upx \
wget \
zip \
zstd \
&& ln -sf /usr/bin/python3 /usr/bin/python \
&& ln -sf /usr/bin/pip3 /usr/bin/pip \
&& rm -rf /var/lib/apt/lists/*

# Runner user
RUN adduser --disabled-password --gecos "" --uid 1000 runner

RUN test -n "$TARGETPLATFORM" || (echo "TARGETPLATFORM must be set" && false)

# Setup subuid and subgid so that "--userns-remap=default" works
RUN set -eux; \
addgroup --system dockremap; \
adduser --system --ingroup dockremap dockremap; \
echo 'dockremap:165536:65536' >> /etc/subuid; \
echo 'dockremap:165536:65536' >> /etc/subgid

ENV RUNNER_ASSETS_DIR=/runnertmp

# Runner download supports amd64 as x64
RUN ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
&& export ARCH \
&& if [ "$ARCH" = "amd64" ]; then export ARCH=x64 ; fi \
&& mkdir -p "$RUNNER_ASSETS_DIR" \
&& cd "$RUNNER_ASSETS_DIR" \
&& curl -L -o runner.tar.gz https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${ARCH}-${RUNNER_VERSION}.tar.gz \
&& tar xzf ./runner.tar.gz \
&& rm runner.tar.gz \
&& ./bin/installdependencies.sh \
&& apt-get install -y libyaml-dev \
&& rm -rf /var/lib/apt/lists/*

RUN echo AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache > /runner.env \
&& mkdir /opt/hostedtoolcache \
&& chgrp runner /opt/hostedtoolcache \
&& chmod g+rwx /opt/hostedtoolcache

# Configure hooks folder structure.
COPY hooks /etc/arc/hooks/

# arch command on OS X reports "i386" for Intel CPUs regardless of bitness
RUN export ARCH=$(echo ${TARGETPLATFORM} | cut -d / -f2) \
&& if [ "$ARCH" = "arm64" ]; then export ARCH=aarch64 ; fi \
&& if [ "$ARCH" = "amd64" ] || [ "$ARCH" = "i386" ]; then export ARCH=x86_64 ; fi \
&& curl -f -L -o /usr/local/bin/dumb-init https://github.com/Yelp/dumb-init/releases/download/v${DUMB_INIT_VERSION}/dumb-init_${DUMB_INIT_VERSION}_${ARCH} \
&& chmod +x /usr/local/bin/dumb-init

COPY entrypoint.sh logger.bash rootless-startup.sh update-status /usr/bin/

RUN chmod +x /usr/bin/rootless-startup.sh /usr/bin/entrypoint.sh

# Make the rootless runner directory executable
RUN mkdir /run/user/1000 \
&& chown runner:runner /run/user/1000 \
&& chmod a+x /run/user/1000

# Add the Python "User Script Directory" to the PATH
ENV PATH="${PATH}:${HOME}/.local/bin:/home/runner/bin"
ENV ImageOS=ubuntu20
ENV DOCKER_HOST=unix:///run/user/1000/docker.sock
ENV XDG_RUNTIME_DIR=/run/user/1000

RUN echo "PATH=${PATH}" > /etc/environment \
&& echo "ImageOS=${ImageOS}" >> /etc/environment \
&& echo "DOCKER_HOST=${DOCKER_HOST}" >> /etc/environment \
&& echo "XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR}" >> /etc/environment

ENV HOME=/home/runner

# No group definition, as that makes it harder to run docker.
USER runner

# Docker installation
ENV SKIP_IPTABLES=1
RUN curl -fsSL https://get.docker.com/rootless | sh

# Docker-compose installation
RUN curl -L "https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-Linux-x86_64" -o /home/runner/bin/docker-compose ; \
chmod +x /home/runner/bin/docker-compose

ENTRYPOINT ["/usr/local/bin/dumb-init", "--"]
CMD ["rootless-startup.sh"]
27 changes: 27 additions & 0 deletions runner/rootless-startup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/bin/bash
source logger.bash

log.notice "Writing out Docker config file"
/bin/bash <<SCRIPT
mkdir -p /home/runner/.config/docker/

if [ ! -f /home/runner/.config/docker/daemon.json ]; then
echo "{}" > /home/runner/.config/docker/daemon.json
fi

if [ -n "${MTU}" ]; then
jq ".\"mtu\" = ${MTU}" /home/runner/.config/docker/daemon.json > /tmp/.daemon.json && mv /tmp/.daemon.json /home/runner/.config/docker/daemon.json
# See https://docs.docker.com/engine/security/rootless/
echo "environment=DOCKERD_ROOTLESS_ROOTLESSKIT_MTU=${MTU}" >> /etc/supervisor/conf.d/dockerd.conf
fi

if [ -n "${DOCKER_REGISTRY_MIRROR}" ]; then
jq ".\"registry-mirrors\"[0] = \"${DOCKER_REGISTRY_MIRROR}\"" /home/runner/.config/docker/daemon.json > /tmp/.daemon.json && mv /tmp/.daemon.json /home/runner/.config/docker/daemon.json
fi
SCRIPT

log.notice "Starting Docker (rootless)"
/home/runner/bin/dockerd-rootless.sh --config-file /home/runner/.config/docker/daemon.json >> /dev/null 2>&1 &

# Wait processes to be running
entrypoint.sh