Run Docker containers in microVM in a Docker container
To build a dockerized CI pipeline, you may have to run docker inside a docker container (e.g., building docker images in a Jenkins container).
However, due to the Docker container's permission, privileges, etc., spawning containers inside a container is a little bit tricky and sacrifices security.
DinD (Docker-in-Docker), for example, requires --privileged flag, and using it should be avoided for security.
DooD (Docker-out-of-Docker), on the other hand, shares the host-side dockerd daemon by binding the host's docker control socket (/var/run/docker.sock).
This method does not require --privileged flag but allows direct access to the host-side dockerd, giving full view and permission of the daemon.
Sysbox is a special container runtime for this purpose; however, it requires host-side setup.
We build DinV (Docker-in-VM), which allows spawning containers in a docker container without --privileged, dockerd sharing, and a special runtime.
DinV uses QEMU's microVM to run a lightweight virtual machine with a separate Linux image (Alpine Linux) and dockerd daemon.
Also, it supports port binding (via hostfwd of SLIRP) and file system sharing (via virtio-9p).
| DooD | DinD | Sysbox | DinV | |
|---|---|---|---|---|
| Requirement | Bind mounting of /var/run/docker.sock |
--privileged flag |
Special runtime (sysbox-runc) |
KVM device option (--device /dev/kvm) |
| Require special runtime | No | No | Yes | No |
dockerd separation |
Shared with host (no separation) | Separated | Separated | Separated |
| Security | Container can create/delete/modify host's containers | Weak isolation (require privileged mode) | ? | Strong isolation (VM isolation + unprivileged) |
| Compute/memory performance | Native | Native | ? | Near-native (VT-x accelerated) |
| Networking performance | Native | Native | ? | Poor (depends on SLIRP) |
| I/O performance | Native | Native | ? | Volumes: Near-native (virtio-blk) Bind mounts: Poor (virtio-9p) |
$ docker run -d --rm --name dinv --device /dev/kvm pusnow/dinv:latest
$ # wait few seconds
$ docker exec -it dinv docker run -it --rm debian
Unable to find image 'debian:latest' locally
latest: Pulling from library/debian
0c6b8ff8c37e: Pull complete
Digest: sha256:fb45fd4e25abe55a656ca69a7bef70e62099b8bb42a279a5e0ea4ae1ab410e0d
Status: Downloaded newer image for debian:latest
root@1ee213376f22:/# Or, you can use Docker management port.
$ docker run -d --rm --name dinv --device /dev/kvm -p127.0.0.1:2375:2375 pusnow/dinv:latest
$ # wait few seconds
$ docker -H tcp://127.0.0.1:2375 ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ DOCKER_HOST="tcp://127.0.0.1:2375" docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES- Note: you have to specifiy forwared ports via
DINV_TCP_PORTSenvironment variable. Semicolon-separated list is allowed. - Note 2: DinV uses a bridged network (172.19.0.0/16) inside VM. Make sure your host docker network does not use the range (Docker's default network range is 172.17.0.0/16).
$ docker run -d --rm -p8080:8080 -e DINV_TCP_PORTS=8080 --name dinv --device /dev/kvm pusnow/dinv:latest
$ # wait few seconds
$ docker exec -it dinv docker run -d -p8080:80 --rm nginx
$ curl http://127.0.0.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...- Note: you have to specifiy bind mounts via
DINV_MOUNTSenvironment variable. Semicolon-separated list is allowed. - Note: we've found virtio-9p is terribly slow. Please, avoid I/O heavy workloads on it.
$ mkdir -p data && echo "Hello world" > data/hello.txt
$ cat data/hello.txt
Hello world
$ docker run -d --rm -v $PWD/data:/data -e DINV_MOUNTS=/data --name dinv --device /dev/kvm pusnow/dinv:latest
$ # wait few seconds
$ docker exec -it dinv docker run -it -v/data:/data debian cat /data/hello.txt
Unable to find image 'debian:latest' locally
latest: Pulling from library/debian
0c6b8ff8c37e: Pull complete
Digest: sha256:fb45fd4e25abe55a656ca69a7bef70e62099b8bb42a279a5e0ea4ae1ab410e0d
Status: Downloaded newer image for debian:latest
Hello world| Name | Description |
|---|---|
DINV_CPUS |
Number of CPUS for VM (default: 1) |
DINV_MEMORY |
Amount of memory for VM (default: 512M) |
DINV_TCP_PORTS |
TCP port numbers for forwarding. Semicolon-separated list (default: none) |
DINV_UDP_PORTS |
UDP port numbers for forwarding. Semicolon-separated list (default: none) |
DINV_MOUNTS |
Paths for bind mounts. Semicolon-separated list (default: none) |
DINV_DOCKER_SIZE |
Initial size of Docker VM disk for /var/lib/docker (default: 64G) |
DINV_VOLUME_PATH |
If specified, DinV mount an additional VM disk image on the path (default: none) |
DINV_VOLUME_SIZE |
Initial size of the DINV_VOLUME_PATH VM disk image |
DINV_VOLUME_UID |
UID of the DINV_VOLUME_PATH |
DINV_VOLUME_GID |
GID of the DINV_VOLUME_PATH |
DINV_DOCKER_SOCK_UID |
UID of the /var/run/docker.sock |
DINV_DOCKER_SOCK_GID |
GID of the /var/run/docker.sock |
DINV_SHUTDOWN_TIMEOUT |
DinV dockerd's shutdown time out value (default: 5) |
DINV_MACHINE |
DinV machine type. Use microvm for a microVM and q35 for a normal VM (default: microvm) |
| Path | Description |
|---|---|
/docker |
VM disk image store for Docker (/var/lib/docker) |
/volume |
VM disk image store for an additioanl DinV volume (DINV_VOLUME_PATH) |
| Port Number | Description |
|---|---|
2375 |
Docker control port |
- virtio-fs (current QEMU microVM does not support it)