Get docker to run inside LXD #608

Closed
benschw opened this Issue May 8, 2015 · 19 comments

Comments

Projects
None yet
@benschw

benschw commented May 8, 2015

Description of problem:

When trying to run Docker in an ubuntu trusty lxc container created with lxd (cli, not openstack) on an ubuntu vivid host (created with vagrant from a cloud-images.ubuntu.com image) I get the following error:

# docker -d --exec-driver=lxc
INFO[0000] +job serveapi(unix:///var/run/docker.sock)   
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
FATA[0000] Shutting down daemon due to errors: error intializing graphdriver: permission denied 

I posted this as a Docker issue first (moby/moby#13054) but didn't get very far. Hoping to find out more here.

From VirtualBox host:

docker version:

Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64

docker info:

Containers: 0
Images: 0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.19.0-15-generic
Operating System: Ubuntu 15.04
CPUs: 1
Total Memory: 489.1 MiB
Name: vagrant-ubuntu-vivid-64
ID: LF2T:DNN2:JF5H:IMQS:S7LK:7WDA:CETE:ABRV:LLKH:APKY:75UI:AT2B
WARNING: No swap limit support

uname -a:

Linux vagrant-ubuntu-vivid-64 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

From lxc container

docker version:

Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
FATA[0000] Get http:///var/run/docker.sock/v1.18/version: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS? 

docker info:

FATA[0000] Get http:///var/run/docker.sock/v1.18/info: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS? 

uname -a:

Linux test 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

  • VirtualBox host (Ubuntu Vivid from cloud-images.ubuntu.com) running lxd
  • the lxc guest is running Ubuntu Trusty

How reproducible:

Steps to Reproduce:

  1. Boot ubuntu vivid vm in VirtualBox
  2. Install Docker on host (to make sure kernel has everything)
    1. wget -qO- https://get.docker.com/ | sh
  3. Install Lxd Container
    1. sudo apt-get install -y lxd
    2. sudo service lxd start
    3. sudo lxd-images import lxc ubuntu trusty amd64 --alias trusty
    4. sudo lxc launch trusty test
    5. sudo lxc exec test -- /bin/bash
  4. Install Docker on lxc guest
    1. apt-get install wget
    2. wget -qO- https://get.docker.com/ | sh
  5. Try to start Docker
    1. service docker start (fails)
    2. docker -d --exec-driver=lxc (fails)
    3. docker -d (fails)

Actual Results:

docker -d --exec-driver=lxc
INFO[0000] +job serveapi(unix:///var/run/docker.sock)   
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
FATA[0000] Shutting down daemon due to errors: error intializing graphdriver: permission denied

Expected Results:

docker -d --exec-driver=lxc
...
INFO[0000] Daemon has completed initialization          

Additional info:

I have tried this without docker installed on the host and with it installed but not running in addition to the above "steps to reproduce."

My thoughts were that any kernel dependencies docker needs would get sucked in on the virtualbox host by installing docker there and would be made available to the lxd guest where I actually want to run docker from.

@stgraber stgraber changed the title from Trouble running docker in a lxc container created with lxd to Get docker to run inside LXD May 8, 2015

@stgraber stgraber added the Feature label May 8, 2015

@stgraber stgraber added this to the lxd-0.14 milestone May 8, 2015

@stgraber

This comment has been minimized.

Show comment
Hide comment
@stgraber

stgraber May 8, 2015

Member

Hi,

Getting Docker to run inside a regular LXD container is on our TODO list.

We don't expect any change to be needed on the LXD side of things, but instead, Docker will have to be taught to play nice with LXD's unprivileged (secure) containers.

It's on our TODO list and I did some tests a week or so ago. It's however difficult to estimate exactly how long it'll take to resolve most issues.

So far those I'm aware of are:

  • Running Docker on something other than btrfs will fail (no planned fixes)
  • Docker needs to be taught to not fail on missing /dev/fuse (or failure to create it)
  • Docker must be fixed to work with LXC 1.1 or higher (in progress)
  • Docker must be tweaked not to fail on unprivileged btrfs (where you can create/destroy subvolumes but not list them)
  • Updates to our apparmor profile to allow Docker to do its thing
  • Update to LXD to recursively clean btrfs volumes
Member

stgraber commented May 8, 2015

Hi,

Getting Docker to run inside a regular LXD container is on our TODO list.

We don't expect any change to be needed on the LXD side of things, but instead, Docker will have to be taught to play nice with LXD's unprivileged (secure) containers.

It's on our TODO list and I did some tests a week or so ago. It's however difficult to estimate exactly how long it'll take to resolve most issues.

So far those I'm aware of are:

  • Running Docker on something other than btrfs will fail (no planned fixes)
  • Docker needs to be taught to not fail on missing /dev/fuse (or failure to create it)
  • Docker must be fixed to work with LXC 1.1 or higher (in progress)
  • Docker must be tweaked not to fail on unprivileged btrfs (where you can create/destroy subvolumes but not list them)
  • Updates to our apparmor profile to allow Docker to do its thing
  • Update to LXD to recursively clean btrfs volumes
@stgraber

This comment has been minimized.

Show comment
Hide comment
@stgraber

stgraber May 8, 2015

Member

We've had some success running Docker in a privileged container with apparmor disabled but doing so bypasses every single security measure we've got in place so isn't something we really want people to do :)

Member

stgraber commented May 8, 2015

We've had some success running Docker in a privileged container with apparmor disabled but doing so bypasses every single security measure we've got in place so isn't something we really want people to do :)

@benschw

This comment has been minimized.

Show comment
Hide comment
@benschw

benschw May 8, 2015

Haha, well that all sounds great I'm glad its on the roadmap. The potential for application and system container compositions is really exciting!

benschw commented May 8, 2015

Haha, well that all sounds great I'm glad its on the roadmap. The potential for application and system container compositions is really exciting!

@ranjib

This comment has been minimized.

Show comment
Hide comment
@ranjib

ranjib May 8, 2015

Contributor

will it be easier to support run app container spec based images under LXC/LXD[1]. Then we can just write lxc rootfs -> app container spec image converters, that will allow running rocket and similar (hopefully but less likely docker :-/ ) app containers.

[1]https://github.com/appc/spec

Contributor

ranjib commented May 8, 2015

will it be easier to support run app container spec based images under LXC/LXD[1]. Then we can just write lxc rootfs -> app container spec image converters, that will allow running rocket and similar (hopefully but less likely docker :-/ ) app containers.

[1]https://github.com/appc/spec

@stgraber

This comment has been minimized.

Show comment
Hide comment
@stgraber

stgraber May 8, 2015

Member

LXD is currently solely focused on running full system containers, we believe there are great container managers for application containers and rather than becoming yet another one of those, we instead want to ensure that they run inside our full containers.

Member

stgraber commented May 8, 2015

LXD is currently solely focused on running full system containers, we believe there are great container managers for application containers and rather than becoming yet another one of those, we instead want to ensure that they run inside our full containers.

@chuckbutler

This comment has been minimized.

Show comment
Hide comment
@chuckbutler

chuckbutler Jul 3, 2015

@stgraber I've done some investigation on this recently - and it appears we still need to lift all apparmor restrictions and set the container to unconstrained. Is this an accurate assessment of where we stand today on this?

@stgraber I've done some investigation on this recently - and it appears we still need to lift all apparmor restrictions and set the container to unconstrained. Is this an accurate assessment of where we stand today on this?

@stgraber

This comment has been minimized.

Show comment
Hide comment
@stgraber

stgraber Jul 3, 2015

Member

@chuckbutler that's where we were last I tried, yes. Frankly, there's not much we can do on the LXD side for this. We'll certainly be offering more security knobs to fine tune the apparmor profile, but we'll need some work on Docker to work inside an unprivileged container.

Member

stgraber commented Jul 3, 2015

@chuckbutler that's where we were last I tried, yes. Frankly, there's not much we can do on the LXD side for this. We'll certainly be offering more security knobs to fine tune the apparmor profile, but we'll need some work on Docker to work inside an unprivileged container.

@stgraber stgraber modified the milestones: lxd-0.14, lxd-0.15 Jul 7, 2015

@stgraber stgraber modified the milestones: lxd-0.15, lxd-0.16 Jul 22, 2015

@stgraber stgraber modified the milestones: lxd-0.16, lxd-0.17 Aug 4, 2015

@stgraber stgraber modified the milestones: lxd-0.17, lxd-0.18 Aug 25, 2015

@stgraber stgraber modified the milestones: lxd-0.18, lxd-0.19, lxd-0.20 Sep 15, 2015

@stgraber stgraber modified the milestones: lxd-0.20, lxd-0.21, meta-1.0 Sep 29, 2015

@stgraber stgraber modified the milestones: meta-1.0, lxd-0.21 Sep 30, 2015

@webwurst

This comment has been minimized.

Show comment
Hide comment
@webwurst

webwurst Dec 5, 2015

Any news on this?

webwurst commented Dec 5, 2015

Any news on this?

@hallyn

This comment has been minimized.

Show comment
Hide comment
@hallyn

hallyn Jan 11, 2016

Member
Member

hallyn commented Jan 11, 2016

@stgraber

This comment has been minimized.

Show comment
Hide comment
@stgraber

stgraber Jan 12, 2016

Member

@hallyn How difficult would it be to add detection of existing apparmor confinement to Docker / OpenContainer and have it ignore profile change failures in that case?

That should fix things a bit for the pre-apparmor namespace world and should also just work once apparmor namespacing lands.

Member

stgraber commented Jan 12, 2016

@hallyn How difficult would it be to add detection of existing apparmor confinement to Docker / OpenContainer and have it ignore profile change failures in that case?

That should fix things a bit for the pre-apparmor namespace world and should also just work once apparmor namespacing lands.

@hallyn

This comment has been minimized.

Show comment
Hide comment
@hallyn

hallyn Jan 12, 2016

Member
Member

hallyn commented Jan 12, 2016

@gaetronik

This comment has been minimized.

Show comment
Hide comment
@gaetronik

gaetronik Jan 21, 2016

I tried to run docker in a privileged container with apparmor disabled but it fails on cgroup mount detection. Where can i find the @hallyn patches to docker to give it a try?

I tried to run docker in a privileged container with apparmor disabled but it fails on cgroup mount detection. Where can i find the @hallyn patches to docker to give it a try?

@stgraber stgraber modified the milestones: meta-2.0, lxd-2.0.0.rc1 Feb 10, 2016

@tebanep

This comment has been minimized.

Show comment
Hide comment

tebanep commented Feb 20, 2016

+1

@stgraber

This comment has been minimized.

Show comment
Hide comment
@stgraber

stgraber Feb 20, 2016

Member

Docker support is still work in progress but it will require you use a very recent Ubuntu kernel from the current development release combined with the Docker packaged in Ubuntu. That's because the changes needed to Docker and the kernel haven't made it to an upstream release of either.

Member

stgraber commented Feb 20, 2016

Docker support is still work in progress but it will require you use a very recent Ubuntu kernel from the current development release combined with the Docker packaged in Ubuntu. That's because the changes needed to Docker and the kernel haven't made it to an upstream release of either.

@stgraber stgraber modified the milestones: lxd-2.0.0.beta4, lxc-2.0.0.rc1 Feb 23, 2016

@stgraber

This comment has been minimized.

Show comment
Hide comment
@stgraber

stgraber Feb 29, 2016

Member

Closing this now as we have the new "docker" profile in LXD which comes with the config bits needed for docker to be happy and Ubuntu 16.04 now has a kernel and LXC that provide a cgroup setup compatible with Docker.

That's not to say that Ubuntu's Docker or even the current upstream release works inside LXD with those. @hallyn has a bunch of patches that are required for that. Those have been sent upstream and for the most part merged, so hopefully an upcoming Docker release will work out of the box.

For Ubuntu users, the version of Docker in the Ubuntu archive at 16.04 release time will include any needed patch so that it works inside LXD out of the box.

Member

stgraber commented Feb 29, 2016

Closing this now as we have the new "docker" profile in LXD which comes with the config bits needed for docker to be happy and Ubuntu 16.04 now has a kernel and LXC that provide a cgroup setup compatible with Docker.

That's not to say that Ubuntu's Docker or even the current upstream release works inside LXD with those. @hallyn has a bunch of patches that are required for that. Those have been sent upstream and for the most part merged, so hopefully an upcoming Docker release will work out of the box.

For Ubuntu users, the version of Docker in the Ubuntu archive at 16.04 release time will include any needed patch so that it works inside LXD out of the box.

@fuzzy76

This comment has been minimized.

Show comment
Hide comment
@fuzzy76

fuzzy76 Jan 30, 2017

Is there any way for me to know which docker release has all patches needed?

fuzzy76 commented Jan 30, 2017

Is there any way for me to know which docker release has all patches needed?

@stgraber

This comment has been minimized.

Show comment
Hide comment
@stgraber

stgraber Jan 30, 2017

Member

@fuzzy76 using docker.io from the Ubuntu repository will always include all the needed patches as Ubuntu gates Docker on working inside LXD (automatic test).

For upstream Docker, 1.13 was reported to work unmodified inside LXD containers. However, Docker upstream has since merged a patch to mitigate for a kernel CVE which interferes with running inside an unprivileged container and so effectively regressed LXD support.

Ubuntu carries a new patch to revert that behavior when a user namespace is detected, but I don't think this made it upstream. Instead a proper kernel fix for the issue is being discussed by the Docker folks.

Member

stgraber commented Jan 30, 2017

@fuzzy76 using docker.io from the Ubuntu repository will always include all the needed patches as Ubuntu gates Docker on working inside LXD (automatic test).

For upstream Docker, 1.13 was reported to work unmodified inside LXD containers. However, Docker upstream has since merged a patch to mitigate for a kernel CVE which interferes with running inside an unprivileged container and so effectively regressed LXD support.

Ubuntu carries a new patch to revert that behavior when a user namespace is detected, but I don't think this made it upstream. Instead a proper kernel fix for the issue is being discussed by the Docker folks.

@timricese

This comment has been minimized.

Show comment
Hide comment
@timricese

timricese Feb 23, 2017

@stgraber do you know if theres an open issue with docker or something so we can follow this? or if theres been any updates in the last 3 weeks?

@stgraber do you know if theres an open issue with docker or something so we can follow this? or if theres been any updates in the last 3 weeks?

@lnussbaum lnussbaum referenced this issue in madynes/distem Jul 25, 2017

Open

docker support is a hack #29

@sqllyw

This comment has been minimized.

Show comment
Hide comment
@sqllyw

sqllyw Oct 5, 2017

any update on this? following this article:
https://stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/
but got error:

Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
ro

sqllyw commented Oct 5, 2017

any update on this? following this article:
https://stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/
but got error:

Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
ro
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment