Add support for unprivileged DinD using Sysbox

[ctalledo]

Up to recently, Docker-in-Docker (DinD) required using privileged containers, which is not ideal since such containers are very insecure and thus compromise host security.

However, there is now a solution to run Docker-in-Docker in a secure (in fact rootless) container: it's called Sysbox, a "next-gen runc" (I am one of the developers).

Sysbox enables users to run VM-workloads inside well isolated rootless containers or pods (e.g., workloads such as systemd, Docker, buildx, KinD, etc.,). It installs easily on Docker hosts or in Kubernetes clusters (see here).

I am thinking that it would be very beneficial to users if the actions-runner-controller could be updated to support alternative runtimes such as Sysbox. This would allow users to run Docker inside Kubernetes pods securely when running GHA jobs.

Deploying a pod with Sysbox simply requires that the pod's spec specify the RuntimeClassName: sysbox-runc plus a user-namespace annotation (more here). The privileged: true flag is no longer required.

Would love to hear your thoughts, and I would be happy to contribute to make this happen.

[bigeasy]

Six hours ago? Wow. I've just started looking through the repository for a first install and I was concerned about the DinD and I found this issue by searching for privileged. I'd be so happy to deploy this immediately and provide feedback, or help in any way I can.

[mumoshu]

@ctalledo Hey! I had been following your awesome sysbox for this exact use-case since two weeks ago and I even included it in my summary at #792 (comment). So- I'm really surprised and glad to see from your timeline that you commented on this project! Thanks for chiming in.

I believe sysbox is the most viable option for us.

Unfortunately, I have not successfully tested it myself, because I'm mainly AWS and EKS shop and EKS doesn't support CRI-O, which is required to use sysbox on K8s as you've kindly explained in nestybox/sysbox#64 (comment).

Anyone reading this thread has access to a CRI-O based Kubernetes cluster and mind helping us to test the solution?

[mumoshu]

Apparently, the relevant KEP is being worked on towards 1.24 kubernetes/enhancements#2101

[ctalledo]

Thanks @mumoshu for your kind words, and congrats on this excellent project too (actions-runner-controller).

You made me realize I need to update Sysbox issue #64 as deploying pods with K8s + Sysbox is now officially supported.

We even created a daemonset that installs CRI-O + Sysbox in the K8s cluster, then reconfigures the Kubelet, and restarts the pods. Installation instructions are here.

This even works on EKS too, so long as you use a Ubuntu Focal (20.04) image for the worker nodes (see here for an example on how to do this).

Thus, it is currently possible for people to use Sysbox on EKS (as well as GKE, AKS, Rancher, and self-managed K8s clusters too).

[ctalledo]

Regarding the relevant KEP you posted, Sysbox is a bit ahead of it (in fact is in "Phase 3 - pods with volumes and inter-pod isolation" already).

Once the KEP catches up, we will adjust Sysbox so that it works with the different modes the KEP proposes.

For now, all is needed is to add an annotation such as io.kubernetes.cri-o.userns-mode: "auto:size=65536" and runtimeClassName: sysbox-runc to the pod spec as shown below:

apiVersion: v1
kind: Pod
metadata:
  name: ubu-bio-systemd-docker
  annotations:
    io.kubernetes.cri-o.userns-mode: "auto:size=65536"
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: ubu-bio-systemd-docker
    image: registry.nestybox.com/nestybox/ubuntu-bionic-systemd-docker
    command: ["/sbin/init"]
  restartPolicy: Never

[jimmyh85]

@mumoshu ,
I would be interested in testing ARC runner pods running on sysbox and I have spun up a cluster with sysbox installed. However, I am lacking the deeper knowledge of what really is required in order to get the solution up and running. I have seen many issues here already talking about this topic but up to now, if I understood correctly we are missing a solution.

Trying to deploy a runnerdeployment on the sysbox node currently fails for me with

  Warning  FailedCreatePodSandBox  39s                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = container create failed: time="2022-03-25T11:16:44Z" level=error msg="container_linux.go:393: starting container process caused: process_linux.go:607: container init caused: rootfs_linux.go:68: setting up rootfs mounts caused: rootfs_linux.go:1104: mounting \"sysfs\" to rootfs \"/var/lib/sysbox/shiftfs/040bab17-9583-4596-bb6d-ed434920f99a\" at \"sys\" caused: operation not permitted"

I guess that sysbox now interferes the installation because it runs as privileged. Any suggestions on how to continue from here?

[ctalledo]

hi @jimmyh85, belated response (missed your comment somehow), but that error you are getting sounds like the pod spec for sysbox is missing the io.kubernetes.cri-o.userns-mode: "auto:size=65536" annotation. This annotation tells CRI-O to launch the pod as "rootless"; without it Sysbox will throw an error as the OCI spec it receives is not quite correct.

It could also be the fact that the pod is being launched as privileged by the actions-runner-controller, though docker run --runtime=sysbox-runc --privileged ... works without problem (the container is not really privileged, it has all caps but it's still rootless).

Hope that helps and apologies again for the belated response.

[shinji62]

@mumoshu Hi, long time no see :)

We are still testing sysbox (EKS) ourself, and would love to integrate with your Kubernetes controllers, but for that to happen we need to remove the privileged flag and be able to add the annotation that @ctalledo share, I don't think this is a huge amount of work, but still we are interested.

I am sure we are not the first one to ask to disable the priv. flag, pretty sure you can kick a quick PR as you know the code better than everyone, or you prefer a PR ?

Thanks
Gwenn

[ajitkumarnayak1976]

we are using EKS optimized OS Images from AWS on Amazon Linux 2, is there support planned for that with this sysbox?

[ctalledo]

I would say: I am not aware of any other solution that allows running Docker (as well as buildx, K8s, k3s, etc.) in unprivileged pods.

[AmadeoRR]

@ctalledo Thanks for the info. SO has sysbox 0.5 been released? I do not see any info on it on the git repo for sysbox.

[ctalledo]

It's 2-3 weeks away, we are working hard at it. The key feature that 0.5 brings is the ability to work on more Linux distros. Prior versions of Sysbox (e.g., <= v0.4.1) work mainly on Ubuntu / Debian.

[renanqts]

Hey guys,

First of all, thanks for the excellent work.
@ctalledo About this topic, do you have any updates?
I tried it out but seems like still has limitations

Sysbox is not supported on this host's distro ("amzn"-2)

[ctalledo]

Hi @renanqts, Nestybox (the startup behind Sysbox) was acquired a few months ago by Docker and that caused some re-prioritization of tasks; as a result Sysbox support for AL2 is not yet completed (plus it requires AL2 to support kernel >= 5.11 and only recently has this been added).

We expect to get some cycles within the next couple of months to add support for this in Sysbox.

[roshvin]

Hi @ctalledo , we are looking for an option to run the DinD with unprivileged option since our CaaS wont allow running pod as root user.
Do you have any working setup for DinD with unprivileged mode.

[ctalledo]

Hi @roshvin, Sysbox will enable you to run Docker inside unprivileged containers (i.e., containers isolated via the user-namespace, where root in the container != unprivileged user at host level). Sysbox runs with Docker or Kubernetes.

Having said this, I don't have a setup with GHA on K8s using Sysbox yet, and this is why I opened this discussion. What's your setup like?

[roshvin]

Hi , thanks for your replay.

1. we have GHE on Prem and enabled the ARC at GHE enterprise level.
2. all features of ARC is working as expected.
3. Our developers do mostly Docker build so in this case the runner is not working since its need a docker engine inside pod.
4. so we are looking for a working solution or a sample runner deployment file.

do you have any sample runnerdeployment file with Sysbox so that i can refer and try.

[ctalledo]

Hi @roshvin, no I don't have such a sample ARC deployment. It sounds as if @mumoshu is making changes that will allow ARC to launch pods with alternative runtimes such as Sysbox, which would then enable users to run Docker inside unprivileged pods.

[mumoshu]

Hi everyone and @ctalledo,
I've recently merged a PR that enhances ARC to enable toggling pod and container's privileged independently of if you enable dind sidecar or not.
#1383
This would allow you to remove privilege from the dind sidecar, which is what you want when you use sysbox as a container runtime.

[shinji62]

I am able to use sysbox for my runner, by using the normal annotations and runtime class for my runner deployment.
I disabled completely docker support for the ARC dockerEnabled: false

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: linux-runner-pre-prod
  namespace: mynamespace
spec:
  template:
    metadata:
      annotations:
        io.kubernetes.cri-o.userns-mode: "auto:size=65536"
    spec:
      image: myrepos/myrunnerimage
      enterprise: test
      group: pre-prod-testing
      labels:
        - self-hosted
        - k8s
        - pre-prod
      env:
        # Tell controller to use --ephemeral vs to be deprecated --once flag.
        - name: RUNNER_FEATURE_FLAG_EPHEMERAL
          value: "true"
        # Delay start to avoid hitting
        # https://github.com/actions-runner-controller/actions-runner-controller/issues/787
        - name: STARTUP_DELAY_IN_SECONDS
          value: "5" 
        # Disables automatic runner updates
        - name: DISABLE_RUNNER_UPDATE
          value: "true"
      runtimeClassName: "sysbox-runc"
      imagePullPolicy: Always
      ephemeral: true
      dockerEnabled: false
      resources:
        limits:
          cpu: "2"
          memory: "4Gi"
        requests:
          cpu: "0.5"
          memory: "1Gi"
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: linux-runner-pre-prod-deployment-autoscaler
  namespace: mynamesapce
spec:
  scaleTargetRef:
    name: linux-runner-pre-prod
  scaleUpTriggers:
  - githubEvent:
      checkRun:
        types: ["created"]
        status: "queued"
    amount: 1
    duration: "5m"
  minReplicas: 1
  maxReplicas: 2

[roshvin]

Hi @shinji62 Thanks for sharing the yaml , we also have same use case . could you pls share which image myrepos/myrunnerimage is this , so that i can download same and test in our environment.

[roshvin]

@shinji62 could you pls help with above query. we are looking for the same solution but couldnt find how can i get / build this image image: myrepos/myrunnerimage

[reiniertimmer]

Just wanted to chip in here as well. We are also running on an (Azure AKS) cluster where privileged containers are not allowed by policy. We just started testing with Sysbox, and it seems to work rather well with the standard images.

A basic RunnerDeployment excerpt would look like this

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
spec:
  template:
    metadata:
      annotations:
        io.kubernetes.cri-o.userns-mode: auto:size=65536
    spec:
      image: "summerwind/actions-runner:latest"
      runtimeClassName: sysbox-runc
      containers:
      - name: docker
        securityContext:
          privileged: false

So the trick for us was to run in regular sidecar mode (with dockerEnabled: true - the default) which would employ the sidecar. However, since that docker sidecar is privileged by default, we had to change that to privileged: false using the custom securityContext setting. Apart from that, setting the sysbox/runc annotations was all that it took.

Using the actions-runner-dind image did not work for us however, as it failed to start with the sysbox runtime (still need to investigate this)

[kevholmes]

Our team was also unable to get sysbox to work with the dind images. The error was rather vague and didn't lead us anywhere helpful. Hopefully will have more time to look at this in the next few months.

[mumoshu]

BTW- Congratulations on the acquisition and kudos to all the hard work you did to make it happen @ctalledo 🎉

https://www.docker.com/blog/docker-advances-container-isolation-and-workloads-with-acquisition-of-nestybox/

I'm definitely looking forward to the future of sysbox!

[ctalledo]

Thank you very much @mumoshu, really appreciate it!

[isarkis]

We ran into nestybox/sysbox#350 with Sysbox set up. It's a blocker for us because our workloads build and run 32 bit binaries. We hope Nestybox can fix this issues sooner rather than later.

[thetredev]

Looks like it's fixed: nestybox/sysbox#789

[mike-chen-samsung]

I am using sysbox with a similar configuration as #977 (reply in thread)

I ran into an issue where if I'm using buildkit with docker build, the build context behaves oddly. For example, if I specify ../ as my context (and change -f appropriately) COPY commands seem to ignore the context, saying the files don't exist. And in other cases, it might say my Dockerfile doesn't exist at the specified path even though it definitely does exist at that path. Things only work fine if I have . as my context.

[ctalledo]

Hi @mike-chen-samsung: if you believe it's a sysbox issue, please open an issue in the Sysbox GitHub repo so we can take a look. Thanks!

[meiswjn]

Sorry for digging this up again - has anyone gotten this to run with the new ARC?
I get "sys\" caused: mount through procfd: operation not permitted"

EDIT: Nevermind, got it working. Now just the nodes need to terminate correctly.