Support docker in hyper container

[ptptptptptpt]

I try to run docker in hyper container, but failed. The problems are:

1. Storage driver
   No storage driver (dm, aufs, overlay, etc...) is supported by hyper kernel.

2. Bridge module

bash-4.1# /sbin/modprobe  -v bridge
install /sbin/modprobe --ignore-install bridge && /sbin/sysctl -q -w net.bridge.bridge-nf-call-arptables=0 net.bridge.bridge-nf-call-iptables=0 net.bridge.bridge-nf-call-ip6tables=0
FATAL: Module bridge not found.
FATAL: Error running install command for bridge
bash-4.1# 

[gao-feng]

after implement #73 , you can compile your own hyper kernel and modules

[wrouesnel]

There's also the issue that the cgroupfs filesystem is not mounted by default. Docker also complains no udev is available:

root@22876dfa0b17:/etc/apt/sources.list.d# docker daemon
INFO[0000] libcontainerd: new containerd process, pid: 4363 
WARN[0001] unable to modify root key limit, number of containers could be limitied by this quota: open /proc/sys/kernel/keys/root_maxkeys: no such file or directory 
ERRO[0001] devmapper: Udev sync is not supported. This will lead to data loss and unexpected behavior. Install a more recent version of libdevmapper or select a different storage driver. For more information, see https://docs.docker.com/engine/reference/commandline/daemon/#daemon-storage-driver-option 
INFO[0001] Graph migration to content-addressability took 0.00 seconds 
WARN[0001] Your kernel does not support cgroup memory limit 
WARN[0001] Unable to find cpu cgroup in mounts          
WARN[0001] Unable to find blkio cgroup in mounts        
WARN[0001] Unable to find cpuset cgroup in mounts       
WARN[0001] mountpoint for pids not found                
FATA[0001] Error starting daemon: Devices cgroup isn't mounted

It seems like it would be nice if hyperstart could run as a non-init process then some of these issues could be more easily solved by pulling in a conventional init.

[wrouesnel]

Ah, finally some success with this.

So I two things are missing: hyperstart needs to mount the cgroup filesystems by default (though you can do it manually). Then it does in fact work, but the overlayfs driver isn't included in the image by default.

[wrouesnel]

Looks like I spoke too soon - there's an issue with CONFIG_KEYS not being set in the kernel which breaks the regular docker OCI runtime.

[wrouesnel]

Looks like you also need CONFIG_EXT4_FS_SECURITY=y to use a block-device backing in for docker.

One of the more bizarre things is I have a ZFS root drive - docker somehow detects it through the 9p filesystem mount, and refuses to use overlay over it.

[gao-feng]

@wrouesnel Thanks for your investigation, we didn't support all the kernel configs needed by docker, since this may be changed, you can use the check script from docker to check the missing configs.

[wrouesnel]

So the last 2 items which prevents this from being close to seamless are that by default hypercontainers do not get a cgroupfs filesystem hierarchy (whereas docker containers do, generally filtered into RO mode unless privileged - not sure if that's even desireable with runv).

The other is that docker hates the 9p filesystem, but it seems like that might be fixed by launching in devicemapper loopback file mode, which effectively handles the ext4 dance I do above, but looks like the modules need to be compiled into hyperstart.

[wrouesnel]

Ok so regarding 9p at least: the problem docker has is when it tries to use it's sqlite3 metadatabase. So all of /var/lib/docker needs to be on a block device to work. This is because 9p sharing by default makes mmap syscalls read-only, and you need to specify cache=mmap to the 9p mount options to enable read-write functionality.

The problem is for certain workloads which don't sync, this can blow up as per https://lwn.net/Articles/571171/

- If you want to read a file after writting in a mmap'd section with
no msync (or msync with MS_ASYNC) and no munmap, you will read what
the servers  knows about -- so the old data. With buffered read,
you'd expect the read to find about the map and use it, we don't.
There is, however, what the documentation says so I don't see this as
a bug. What could probably be done would be to check on read if we
have dirty mapped pages and flush them before reading? I'm not too
sure I like this idea.
 - Likewise, if you write with a map open, the map won't be updated
unless something else invalidates it. I guess we should do that like
the direct write function does.

So superficially this makes docker work "out of the box" with the right kernel. But I'm not convinced this is a "safe" default - i.e. I'm okay with it because the specific runv I have will be running build-servers, but in a general case this could lead to surprising behavior.

Re: cgroups - I've implemented a mountpoint for them in my fork of hyperstart, and it seems to work great. The exact mount semantics are probably worth discussing.