Get docker to run inside LXD

[benschw]

Description of problem:

When trying to run Docker in an ubuntu trusty lxc container created with lxd (cli, not openstack) on an ubuntu vivid host (created with vagrant from a cloud-images.ubuntu.com image) I get the following error:

# docker -d --exec-driver=lxc
INFO[0000] +job serveapi(unix:///var/run/docker.sock)   
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
FATA[0000] Shutting down daemon due to errors: error intializing graphdriver: permission denied 

I posted this as a Docker issue first (moby/moby#13054) but didn't get very far. Hoping to find out more here.

From VirtualBox host:

docker version:

Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64

docker info:

Containers: 0
Images: 0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.19.0-15-generic
Operating System: Ubuntu 15.04
CPUs: 1
Total Memory: 489.1 MiB
Name: vagrant-ubuntu-vivid-64
ID: LF2T:DNN2:JF5H:IMQS:S7LK:7WDA:CETE:ABRV:LLKH:APKY:75UI:AT2B
WARNING: No swap limit support

uname -a:

Linux vagrant-ubuntu-vivid-64 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

From lxc container

docker version:

Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
FATA[0000] Get http:///var/run/docker.sock/v1.18/version: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS? 

docker info:

FATA[0000] Get http:///var/run/docker.sock/v1.18/info: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS? 

uname -a:

Linux test 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

• VirtualBox host (Ubuntu Vivid from cloud-images.ubuntu.com) running lxd
• the lxc guest is running Ubuntu Trusty

How reproducible:

Steps to Reproduce:

1. Boot ubuntu vivid vm in VirtualBox
2. Install Docker on host (to make sure kernel has everything)
   1. wget -qO- https://get.docker.com/ | sh
3. Install Lxd Container
   1. sudo apt-get install -y lxd
   2. sudo service lxd start
   3. sudo lxd-images import lxc ubuntu trusty amd64 --alias trusty
   4. sudo lxc launch trusty test
   5. sudo lxc exec test -- /bin/bash
4. Install Docker on lxc guest
   1. apt-get install wget
   2. wget -qO- https://get.docker.com/ | sh
5. Try to start Docker
   1. service docker start (fails)
   2. docker -d --exec-driver=lxc (fails)
   3. docker -d (fails)

Actual Results:

docker -d --exec-driver=lxc
INFO[0000] +job serveapi(unix:///var/run/docker.sock)   
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
FATA[0000] Shutting down daemon due to errors: error intializing graphdriver: permission denied

Expected Results:

docker -d --exec-driver=lxc
...
INFO[0000] Daemon has completed initialization          

Additional info:

I have tried this without docker installed on the host and with it installed but not running in addition to the above "steps to reproduce."

My thoughts were that any kernel dependencies docker needs would get sucked in on the virtualbox host by installing docker there and would be made available to the lxd guest where I actually want to run docker from.

[stgraber]

Hi,

Getting Docker to run inside a regular LXD container is on our TODO list.

We don't expect any change to be needed on the LXD side of things, but instead, Docker will have to be taught to play nice with LXD's unprivileged (secure) containers.

It's on our TODO list and I did some tests a week or so ago. It's however difficult to estimate exactly how long it'll take to resolve most issues.

So far those I'm aware of are:

• Running Docker on something other than btrfs will fail (no planned fixes)
• Docker needs to be taught to not fail on missing /dev/fuse (or failure to create it)
• Docker must be fixed to work with LXC 1.1 or higher (in progress)
• Docker must be tweaked not to fail on unprivileged btrfs (where you can create/destroy subvolumes but not list them)
• Updates to our apparmor profile to allow Docker to do its thing
• Update to LXD to recursively clean btrfs volumes

[stgraber]

Hi,

Getting Docker to run inside a regular LXD container is on our TODO list.

We don't expect any change to be needed on the LXD side of things, but instead, Docker will have to be taught to play nice with LXD's unprivileged (secure) containers.

It's on our TODO list and I did some tests a week or so ago. It's however difficult to estimate exactly how long it'll take to resolve most issues.

So far those I'm aware of are:

• Running Docker on something other than btrfs will fail (no planned fixes)
• Docker needs to be taught to not fail on missing /dev/fuse (or failure to create it)
• Docker must be fixed to work with LXC 1.1 or higher (in progress)
• Docker must be tweaked not to fail on unprivileged btrfs (where you can create/destroy subvolumes but not list them)
• Updates to our apparmor profile to allow Docker to do its thing
• Update to LXD to recursively clean btrfs volumes

[stgraber]

We've had some success running Docker in a privileged container with apparmor disabled but doing so bypasses every single security measure we've got in place so isn't something we really want people to do :)

[stgraber]

We've had some success running Docker in a privileged container with apparmor disabled but doing so bypasses every single security measure we've got in place so isn't something we really want people to do :)

[benschw]

Haha, well that all sounds great I'm glad its on the roadmap. The potential for application and system container compositions is really exciting!

[benschw]

Haha, well that all sounds great I'm glad its on the roadmap. The potential for application and system container compositions is really exciting!

[ranjib]

will it be easier to support run app container spec based images under LXC/LXD[1]. Then we can just write lxc rootfs -> app container spec image converters, that will allow running rocket and similar (hopefully but less likely docker :-/ ) app containers.

[1]https://github.com/appc/spec

[ranjib]

will it be easier to support run app container spec based images under LXC/LXD[1]. Then we can just write lxc rootfs -> app container spec image converters, that will allow running rocket and similar (hopefully but less likely docker :-/ ) app containers.

[1]https://github.com/appc/spec

[stgraber]

LXD is currently solely focused on running full system containers, we believe there are great container managers for application containers and rather than becoming yet another one of those, we instead want to ensure that they run inside our full containers.

[stgraber]

LXD is currently solely focused on running full system containers, we believe there are great container managers for application containers and rather than becoming yet another one of those, we instead want to ensure that they run inside our full containers.

[chuckbutler]

@stgraber I've done some investigation on this recently - and it appears we still need to lift all apparmor restrictions and set the container to unconstrained. Is this an accurate assessment of where we stand today on this?

[chuckbutler]

@stgraber I've done some investigation on this recently - and it appears we still need to lift all apparmor restrictions and set the container to unconstrained. Is this an accurate assessment of where we stand today on this?

[stgraber]

@chuckbutler that's where we were last I tried, yes. Frankly, there's not much we can do on the LXD side for this. We'll certainly be offering more security knobs to fine tune the apparmor profile, but we'll need some work on Docker to work inside an unprivileged container.

[stgraber]

@chuckbutler that's where we were last I tried, yes. Frankly, there's not much we can do on the LXD side for this. We'll certainly be offering more security knobs to fine tune the apparmor profile, but we'll need some work on Docker to work inside an unprivileged container.

[webwurst]

Any news on this?

[webwurst]

Any news on this?

[hallyn]

Yes, slowly. I've proposed a few commits to opencontainer/docker. When you build these with my patches, run a cgroup-namespaces kernel, and bind mount a file reading '0' over /sys/module/apparmor/paramaters/enabled, docker with the overlay driver works. (There is a bug in the unprivileged overlay code which currently requires you to remove /dev/shm in the parent image, but that will hopefully be fixed soon). It's coming together.

[hallyn]

Yes, slowly. I've proposed a few commits to opencontainer/docker. When you build these with my patches, run a cgroup-namespaces kernel, and bind mount a file reading '0' over /sys/module/apparmor/paramaters/enabled, docker with the overlay driver works. (There is a bug in the unprivileged overlay code which currently requires you to remove /dev/shm in the parent image, but that will hopefully be fixed soon). It's coming together.

[stgraber]

@hallyn How difficult would it be to add detection of existing apparmor confinement to Docker / OpenContainer and have it ignore profile change failures in that case?

That should fix things a bit for the pre-apparmor namespace world and should also just work once apparmor namespacing lands.

[stgraber]

@hallyn How difficult would it be to add detection of existing apparmor confinement to Docker / OpenContainer and have it ignore profile change failures in that case?

That should fix things a bit for the pre-apparmor namespace world and should also just work once apparmor namespacing lands.

[hallyn]

I haven't sufficiently looked into it, but it shouldn't be too hard. Thinking runc will start with a apparmor.AmConfinedAndLocked() fn which docker will use in several places. I don't know whether there will be complications/objections about intermediate domains.

[hallyn]

I haven't sufficiently looked into it, but it shouldn't be too hard. Thinking runc will start with a apparmor.AmConfinedAndLocked() fn which docker will use in several places. I don't know whether there will be complications/objections about intermediate domains.

[gaetronik]

I tried to run docker in a privileged container with apparmor disabled but it fails on cgroup mount detection. Where can i find the @hallyn patches to docker to give it a try?

[gaetronik]

I tried to run docker in a privileged container with apparmor disabled but it fails on cgroup mount detection. Where can i find the @hallyn patches to docker to give it a try?

[tebanep]

+1

[tebanep]

+1

[stgraber]

Docker support is still work in progress but it will require you use a very recent Ubuntu kernel from the current development release combined with the Docker packaged in Ubuntu. That's because the changes needed to Docker and the kernel haven't made it to an upstream release of either.

[stgraber]

Docker support is still work in progress but it will require you use a very recent Ubuntu kernel from the current development release combined with the Docker packaged in Ubuntu. That's because the changes needed to Docker and the kernel haven't made it to an upstream release of either.

[stgraber]

Closing this now as we have the new "docker" profile in LXD which comes with the config bits needed for docker to be happy and Ubuntu 16.04 now has a kernel and LXC that provide a cgroup setup compatible with Docker.

That's not to say that Ubuntu's Docker or even the current upstream release works inside LXD with those. @hallyn has a bunch of patches that are required for that. Those have been sent upstream and for the most part merged, so hopefully an upcoming Docker release will work out of the box.

For Ubuntu users, the version of Docker in the Ubuntu archive at 16.04 release time will include any needed patch so that it works inside LXD out of the box.

[stgraber]

Closing this now as we have the new "docker" profile in LXD which comes with the config bits needed for docker to be happy and Ubuntu 16.04 now has a kernel and LXC that provide a cgroup setup compatible with Docker.

That's not to say that Ubuntu's Docker or even the current upstream release works inside LXD with those. @hallyn has a bunch of patches that are required for that. Those have been sent upstream and for the most part merged, so hopefully an upcoming Docker release will work out of the box.

For Ubuntu users, the version of Docker in the Ubuntu archive at 16.04 release time will include any needed patch so that it works inside LXD out of the box.

[fuzzy76]

Is there any way for me to know which docker release has all patches needed?

[fuzzy76]

Is there any way for me to know which docker release has all patches needed?

[stgraber]

@fuzzy76 using docker.io from the Ubuntu repository will always include all the needed patches as Ubuntu gates Docker on working inside LXD (automatic test).

For upstream Docker, 1.13 was reported to work unmodified inside LXD containers. However, Docker upstream has since merged a patch to mitigate for a kernel CVE which interferes with running inside an unprivileged container and so effectively regressed LXD support.

Ubuntu carries a new patch to revert that behavior when a user namespace is detected, but I don't think this made it upstream. Instead a proper kernel fix for the issue is being discussed by the Docker folks.

[stgraber]

@fuzzy76 using docker.io from the Ubuntu repository will always include all the needed patches as Ubuntu gates Docker on working inside LXD (automatic test).

For upstream Docker, 1.13 was reported to work unmodified inside LXD containers. However, Docker upstream has since merged a patch to mitigate for a kernel CVE which interferes with running inside an unprivileged container and so effectively regressed LXD support.

Ubuntu carries a new patch to revert that behavior when a user namespace is detected, but I don't think this made it upstream. Instead a proper kernel fix for the issue is being discussed by the Docker folks.

[timricese]

@stgraber do you know if theres an open issue with docker or something so we can follow this? or if theres been any updates in the last 3 weeks?

[timricese]

@stgraber do you know if theres an open issue with docker or something so we can follow this? or if theres been any updates in the last 3 weeks?

[sqllyw]

any update on this? following this article:
https://stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/
but got error:

Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
ro

[sqllyw]

any update on this? following this article:
https://stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/
but got error:

Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
ro