Skip to content

Pusnow/dinv

Repository files navigation

dinv (Docker in VM)

Run Docker containers in microVM in a Docker container

Why dinv?

To build a dockerized CI pipeline, you may have to run docker inside a docker container (e.g., building docker images in a Jenkins container). However, due to the Docker container's permission, privileges, etc., spawning containers inside a container is a little bit tricky and sacrifices security. DinD (Docker-in-Docker), for example, requires --privileged flag, and using it should be avoided for security. DooD (Docker-out-of-Docker), on the other hand, shares the host-side dockerd daemon by binding the host's docker control socket (/var/run/docker.sock). This method does not require --privileged flag but allows direct access to the host-side dockerd, giving full view and permission of the daemon. Sysbox is a special container runtime for this purpose; however, it requires host-side setup.

We build DinV (Docker-in-VM), which allows spawning containers in a docker container without --privileged, dockerd sharing, and a special runtime. DinV uses QEMU's microVM to run a lightweight virtual machine with a separate Linux image (Alpine Linux) and dockerd daemon. Also, it supports port binding (via hostfwd of SLIRP) and file system sharing (via virtio-9p).

DooD DinD Sysbox DinV
Requirement Bind mounting of /var/run/docker.sock --privileged flag Special runtime (sysbox-runc) KVM device option (--device /dev/kvm)
Require special runtime No No Yes No
dockerd separation Shared with host (no separation) Separated Separated Separated
Security Container can create/delete/modify host's containers Weak isolation (require privileged mode) ? Strong isolation (VM isolation + unprivileged)
Compute/memory performance Native Native ? Near-native (VT-x accelerated)
Networking performance Native Native ? Poor (depends on SLIRP)
I/O performance Native Native ? Volumes: Near-native (virtio-blk)
Bind mounts: Poor (virtio-9p)

Usage

Basic Usage

$ docker run -d --rm --name dinv --device /dev/kvm pusnow/dinv:latest
$ # wait few seconds
$ docker exec -it dinv docker run -it --rm debian
Unable to find image 'debian:latest' locally
latest: Pulling from library/debian
0c6b8ff8c37e: Pull complete 
Digest: sha256:fb45fd4e25abe55a656ca69a7bef70e62099b8bb42a279a5e0ea4ae1ab410e0d
Status: Downloaded newer image for debian:latest
root@1ee213376f22:/# 

Or, you can use Docker management port.

$ docker run -d --rm --name dinv --device /dev/kvm -p127.0.0.1:2375:2375 pusnow/dinv:latest
$ # wait few seconds
$ docker -H tcp://127.0.0.1:2375 ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
$ DOCKER_HOST="tcp://127.0.0.1:2375" docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Port Forwarding

  • Note: you have to specifiy forwared ports via DINV_TCP_PORTS environment variable. Semicolon-separated list is allowed.
  • Note 2: DinV uses a bridged network (172.19.0.0/16) inside VM. Make sure your host docker network does not use the range (Docker's default network range is 172.17.0.0/16).
$ docker run -d --rm -p8080:8080 -e DINV_TCP_PORTS=8080 --name dinv --device /dev/kvm pusnow/dinv:latest
$ # wait few seconds
$ docker exec -it dinv docker run -d -p8080:80 --rm nginx
$ curl http://127.0.0.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

Bind Mount

  • Note: you have to specifiy bind mounts via DINV_MOUNTS environment variable. Semicolon-separated list is allowed.
  • Note: we've found virtio-9p is terribly slow. Please, avoid I/O heavy workloads on it.
$ mkdir -p data && echo "Hello world" > data/hello.txt
$ cat data/hello.txt
Hello world
$ docker run -d --rm -v $PWD/data:/data -e DINV_MOUNTS=/data --name dinv --device /dev/kvm pusnow/dinv:latest
$ # wait few seconds
$ docker exec -it dinv docker run -it -v/data:/data debian cat /data/hello.txt
Unable to find image 'debian:latest' locally
latest: Pulling from library/debian
0c6b8ff8c37e: Pull complete 
Digest: sha256:fb45fd4e25abe55a656ca69a7bef70e62099b8bb42a279a5e0ea4ae1ab410e0d
Status: Downloaded newer image for debian:latest
Hello world

Environment Variables

Name Description
DINV_CPUS Number of CPUS for VM (default: 1)
DINV_MEMORY Amount of memory for VM (default: 512M)
DINV_TCP_PORTS TCP port numbers for forwarding. Semicolon-separated list (default: none)
DINV_UDP_PORTS UDP port numbers for forwarding. Semicolon-separated list (default: none)
DINV_MOUNTS Paths for bind mounts. Semicolon-separated list (default: none)
DINV_DOCKER_SIZE Initial size of Docker VM disk for /var/lib/docker (default: 64G)
DINV_VOLUME_PATH If specified, DinV mount an additional VM disk image on the path (default: none)
DINV_VOLUME_SIZE Initial size of the DINV_VOLUME_PATH VM disk image
DINV_VOLUME_UID UID of the DINV_VOLUME_PATH
DINV_VOLUME_GID GID of the DINV_VOLUME_PATH
DINV_DOCKER_SOCK_UID UID of the /var/run/docker.sock
DINV_DOCKER_SOCK_GID GID of the /var/run/docker.sock
DINV_SHUTDOWN_TIMEOUT DinV dockerd's shutdown time out value (default: 5)
DINV_MACHINE DinV machine type. Use microvm for a microVM and q35 for a normal VM (default: microvm)

Volumes

Path Description
/docker VM disk image store for Docker (/var/lib/docker)
/volume VM disk image store for an additioanl DinV volume (DINV_VOLUME_PATH)

Ports

Port Number Description
2375 Docker control port

TODO

  • virtio-fs (current QEMU microVM does not support it)

Releases

No releases published

Packages

No packages published