New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

privileged containers get unintentionally constrained by apparmor #5490

Closed
ibuildthecloud opened this Issue Apr 30, 2014 · 21 comments

Comments

Projects
None yet
10 participants
@ibuildthecloud
Contributor

ibuildthecloud commented Apr 30, 2014

For normal containers docker will put the container in the docker-default profile. For privileged containers, no profile is applied and the process is unconfined. Since the container is unconfined, the child processes are subject to getting a profile from the host auto applied base on the binaries path.

To see this happen, on ubuntu, install tcpdump and apparmor on the host. Then install tcpdump in a privileged container. tcpdump in the privileged container will not run.

It seems the correct fix would be to create a "docker-unconfined" profile that is applied to privileged containers. The profile would not actually restrict anything, but allow all. The only purpose of using this profile is to prevent the host from auto applying profiles to container processes.

@crosbymichael

This comment has been minimized.

Show comment
Hide comment
@crosbymichael

crosbymichael Apr 30, 2014

Contributor

@ibuildthecloud Would you like to work on this profile? Another option would be to allow the user to create and apply their own profiles to container via a configuration option.

Contributor

crosbymichael commented Apr 30, 2014

@ibuildthecloud Would you like to work on this profile? Another option would be to allow the user to create and apply their own profiles to container via a configuration option.

@unclejack unclejack changed the title from priviledges containers get unintentionally constrained by apparmor to privileged containers get unintentionally constrained by apparmor May 16, 2014

@bprodoehl

This comment has been minimized.

Show comment
Hide comment
@bprodoehl

bprodoehl Jul 16, 2014

@ibuildthecloud Where did you end up on this issue? I'm in much the same position, with tcpdump and dhclient failing in privileged containers with this:

dhclient: error while loading shared libraries: libc.so.6: cannot open shared object file: Permission denied

While on the Ubuntu 14.04 host, dmesg shows this:

[ 3256.689120] type=1400 audit(1405454041.341:73): apparmor="DENIED" operation="open" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/424cfe04a1a3bf60f5b3ebd2a8f77c636533bf4394b3eb260562d9555a6feeeb/etc/ld.so.cache" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 3256.689322] type=1400 audit(1405454041.341:74): apparmor="DENIED" operation="open" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/5e66087f3ffe002664507d225d07b6929843c3f0299f5335a70c1727c8833737/lib/x86_64-linux-gnu/libc-2.19.so" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 3256.689366] type=1400 audit(1405454041.341:75): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/37fd0624106725eab0680c35a71b39b05f4760879be1999557efc5f9e2d078ce/lib/x86_64-linux-gnu" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 3256.689578] type=1400 audit(1405454041.341:76): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/424cfe04a1a3bf60f5b3ebd2a8f77c636533bf4394b3eb260562d9555a6feeeb/usr/lib/x86_64-linux-gnu" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 3256.689789] type=1400 audit(1405454041.341:77): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/37fd0624106725eab0680c35a71b39b05f4760879be1999557efc5f9e2d078ce/lib" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 3256.689979] type=1400 audit(1405454041.341:78): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/424cfe04a1a3bf60f5b3ebd2a8f77c636533bf4394b3eb260562d9555a6feeeb/usr/lib" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

I guess I want a similar "docker-unconfined" profile for privileged containers, but I'm not all that familiar with AppArmor.

bprodoehl commented Jul 16, 2014

@ibuildthecloud Where did you end up on this issue? I'm in much the same position, with tcpdump and dhclient failing in privileged containers with this:

dhclient: error while loading shared libraries: libc.so.6: cannot open shared object file: Permission denied

While on the Ubuntu 14.04 host, dmesg shows this:

[ 3256.689120] type=1400 audit(1405454041.341:73): apparmor="DENIED" operation="open" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/424cfe04a1a3bf60f5b3ebd2a8f77c636533bf4394b3eb260562d9555a6feeeb/etc/ld.so.cache" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 3256.689322] type=1400 audit(1405454041.341:74): apparmor="DENIED" operation="open" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/5e66087f3ffe002664507d225d07b6929843c3f0299f5335a70c1727c8833737/lib/x86_64-linux-gnu/libc-2.19.so" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 3256.689366] type=1400 audit(1405454041.341:75): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/37fd0624106725eab0680c35a71b39b05f4760879be1999557efc5f9e2d078ce/lib/x86_64-linux-gnu" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 3256.689578] type=1400 audit(1405454041.341:76): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/424cfe04a1a3bf60f5b3ebd2a8f77c636533bf4394b3eb260562d9555a6feeeb/usr/lib/x86_64-linux-gnu" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 3256.689789] type=1400 audit(1405454041.341:77): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/37fd0624106725eab0680c35a71b39b05f4760879be1999557efc5f9e2d078ce/lib" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 3256.689979] type=1400 audit(1405454041.341:78): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/sbin/dhclient" name="var/lib/docker/aufs/diff/424cfe04a1a3bf60f5b3ebd2a8f77c636533bf4394b3eb260562d9555a6feeeb/usr/lib" pid=5457 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

I guess I want a similar "docker-unconfined" profile for privileged containers, but I'm not all that familiar with AppArmor.

@bprodoehl

This comment has been minimized.

Show comment
Hide comment
@bprodoehl

bprodoehl Jul 16, 2014

One workaround I've found is to relocate the affected binaries in the container so they don't match where they are on the host. So /sbin/dhclient in the container moves to /usr/sbin/dhclient, and /usr/sbin/tcpdump in the container moves to /usr/bin/tcpdump, for instance, and then both are fine. I'd much rather have a real fix for the problem, but short of that, this seems to work.

bprodoehl commented Jul 16, 2014

One workaround I've found is to relocate the affected binaries in the container so they don't match where they are on the host. So /sbin/dhclient in the container moves to /usr/sbin/dhclient, and /usr/sbin/tcpdump in the container moves to /usr/bin/tcpdump, for instance, and then both are fine. I'd much rather have a real fix for the problem, but short of that, this seems to work.

@crosbymichael

This comment has been minimized.

Show comment
Hide comment
@crosbymichael

crosbymichael Jul 16, 2014

Contributor

For immediate issue I would recommend not installing apparmor profiles on your host system if you don't want them to be applied to the processes running in containers.

As for a fix in docker I you be interested in discussing a flag --apparmor-profile so that you can create and set your own profiles for a docker run.

Contributor

crosbymichael commented Jul 16, 2014

For immediate issue I would recommend not installing apparmor profiles on your host system if you don't want them to be applied to the processes running in containers.

As for a fix in docker I you be interested in discussing a flag --apparmor-profile so that you can create and set your own profiles for a docker run.

@scadgek

This comment has been minimized.

Show comment
Hide comment
@scadgek

scadgek Aug 29, 2014

But what should I do if I want to use AppArmor inside Docker container?

  1. Without AppArmor installed on host Docker daemon won't start
  2. If I unload all AppArmor profiles on host and then run service apparmor reload in docker, the profiles on my host also reload and I end up getting nodejs: error while loading shared libraries: libz.so.1: cannot open shared object file: Permission denied
    Note that I don't have any profile for nodejs on my host, only inside docker container.

scadgek commented Aug 29, 2014

But what should I do if I want to use AppArmor inside Docker container?

  1. Without AppArmor installed on host Docker daemon won't start
  2. If I unload all AppArmor profiles on host and then run service apparmor reload in docker, the profiles on my host also reload and I end up getting nodejs: error while loading shared libraries: libz.so.1: cannot open shared object file: Permission denied
    Note that I don't have any profile for nodejs on my host, only inside docker container.
@bprodoehl

This comment has been minimized.

Show comment
Hide comment
@bprodoehl

bprodoehl Aug 29, 2014

I never found a better solution than moving the binaries in the guest so that they weren't at the same path as the host, and I was also seeing it with binaries that didn't appear to have their own profile (like dhclient did appear to have its own profile, but tcpdump did not, and both were blocked by AppArmor on the host).

If you want to use an AppArmor profile for nodejs in your guest, could you move the nodejs binary to a different path and have your profile in the guest apply to that location?

I presume that you're seeing this only in a privileged container. I've been anxious to go back and try replacing --privileged with the appropriate --cap-add and --cap-drop options to see if that also eases this AppArmor nonsense.

bprodoehl commented Aug 29, 2014

I never found a better solution than moving the binaries in the guest so that they weren't at the same path as the host, and I was also seeing it with binaries that didn't appear to have their own profile (like dhclient did appear to have its own profile, but tcpdump did not, and both were blocked by AppArmor on the host).

If you want to use an AppArmor profile for nodejs in your guest, could you move the nodejs binary to a different path and have your profile in the guest apply to that location?

I presume that you're seeing this only in a privileged container. I've been anxious to go back and try replacing --privileged with the appropriate --cap-add and --cap-drop options to see if that also eases this AppArmor nonsense.

@scadgek

This comment has been minimized.

Show comment
Hide comment
@scadgek

scadgek Aug 29, 2014

I moved /usr/bin/nodejs to /usr/sbin/nodejs and created corresponding profile. No help - same error after service apparmor reload.

I don't really know how to use --cap-add and --cap-drop, I mean, what permission should I add or drop? The only reason why I use --privileged is that if I don't I get an error on service apparmor reload (see http://stackoverflow.com/questions/25533666/cannot-reload-or-start-apparmor-in-docker )

scadgek commented Aug 29, 2014

I moved /usr/bin/nodejs to /usr/sbin/nodejs and created corresponding profile. No help - same error after service apparmor reload.

I don't really know how to use --cap-add and --cap-drop, I mean, what permission should I add or drop? The only reason why I use --privileged is that if I don't I get an error on service apparmor reload (see http://stackoverflow.com/questions/25533666/cannot-reload-or-start-apparmor-in-docker )

@bprodoehl

This comment has been minimized.

Show comment
Hide comment
@bprodoehl

bprodoehl Aug 29, 2014

That's odd that you're seeing it even after moving the binary. What is dmesg on your host saying? Or however you prefer to debug AppArmor, if dmesg isn't your thing. It looks like the capability you'd need based on what I'm seeing in that SO question is CAP_SYS_ADMIN, which looks like a giant grab-bag of administrator stuff. That makes me think that you'd see the same problems if you switched from --privileged to --cap-add. The capabilities are listed here: http://man7.org/linux/man-pages/man7/capabilities.7.html

bprodoehl commented Aug 29, 2014

That's odd that you're seeing it even after moving the binary. What is dmesg on your host saying? Or however you prefer to debug AppArmor, if dmesg isn't your thing. It looks like the capability you'd need based on what I'm seeing in that SO question is CAP_SYS_ADMIN, which looks like a giant grab-bag of administrator stuff. That makes me think that you'd see the same problems if you switched from --privileged to --cap-add. The capabilities are listed here: http://man7.org/linux/man-pages/man7/capabilities.7.html

@crosbymichael

This comment has been minimized.

Show comment
Hide comment
@crosbymichael

crosbymichael Aug 29, 2014

Contributor

Why did you install the profiles on your host in the first place? What was the need?

Contributor

crosbymichael commented Aug 29, 2014

Why did you install the profiles on your host in the first place? What was the need?

@bprodoehl

This comment has been minimized.

Show comment
Hide comment
@bprodoehl

bprodoehl Aug 29, 2014

In my case, I didn't do anything special. Bog standard install of Ubuntu 14.04 and Docker, install tcpdump on the host, run a privileged ubuntu container, install tcpdump in the container, try to run tcpdump in the container, and AppArmor on the host rejects it.

bprodoehl commented Aug 29, 2014

In my case, I didn't do anything special. Bog standard install of Ubuntu 14.04 and Docker, install tcpdump on the host, run a privileged ubuntu container, install tcpdump in the container, try to run tcpdump in the container, and AppArmor on the host rejects it.

@scadgek

This comment has been minimized.

Show comment
Hide comment
@scadgek

scadgek Aug 29, 2014

2bprodoehl
I don't know how to debug AppArmor yet, but I'll try some time later and respond you on the results. And thanks for explanations about cap-add.

2crosbymichael
I didn't install any profiles on my host, if you were talking to me.

scadgek commented Aug 29, 2014

2bprodoehl
I don't know how to debug AppArmor yet, but I'll try some time later and respond you on the results. And thanks for explanations about cap-add.

2crosbymichael
I didn't install any profiles on my host, if you were talking to me.

@jessfraz

This comment has been minimized.

Show comment
Hide comment
@jessfraz

jessfraz Feb 25, 2015

Contributor

@ibuildthecloud is this still an issue, did we end up fixing the profile?

Contributor

jessfraz commented Feb 25, 2015

@ibuildthecloud is this still an issue, did we end up fixing the profile?

@Jamlee

This comment has been minimized.

Show comment
Hide comment
@Jamlee

Jamlee Apr 12, 2015

@ibuildthecloud @jfrazelle could we solve this issue on release 1.6?

Jamlee commented Apr 12, 2015

@ibuildthecloud @jfrazelle could we solve this issue on release 1.6?

@ewindisch

This comment has been minimized.

Show comment
Hide comment
@ewindisch

ewindisch May 15, 2015

Contributor

I've put together a change for this, although I need to test it before submitting the PR:
ewindisch@d45be4e

Contributor

ewindisch commented May 15, 2015

I've put together a change for this, although I need to test it before submitting the PR:
ewindisch@d45be4e

@ewindisch

This comment has been minimized.

Show comment
Hide comment
@ewindisch

ewindisch May 15, 2015

Contributor

#dibs

Contributor

ewindisch commented May 15, 2015

#dibs

@bprodoehl

This comment has been minimized.

Show comment
Hide comment
@bprodoehl

bprodoehl May 27, 2015

@ewindisch any progress on testing your change for this? I'm a bit stuck trying to get NetworkManager to run in a u14.04 container on a u14.04 host due to this problem, so I'm trying out your change right now. I'll let you know how it goes.

bprodoehl commented May 27, 2015

@ewindisch any progress on testing your change for this? I'm a bit stuck trying to get NetworkManager to run in a u14.04 container on a u14.04 host due to this problem, so I'm trying out your change right now. I'll let you know how it goes.

@bprodoehl

This comment has been minimized.

Show comment
Hide comment
@bprodoehl

bprodoehl May 27, 2015

The patches from @ewindisch don't seem quite right anymore, and I'm out of sync on where AppArmor things are moving. I did manage to get around my problem by applying his one-line change to have privileged containers use the "docker-unconfined" profile, rather than "unconfined", and then I put this profile in /etc/apparmor.d/docker-unconfined:

#include <tunables/global>


profile docker-unconfined flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>
  network,
  capability,
  file,
  umount,
}

I don't know all the implications, but that's what finally worked for me last night.

bprodoehl commented May 27, 2015

The patches from @ewindisch don't seem quite right anymore, and I'm out of sync on where AppArmor things are moving. I did manage to get around my problem by applying his one-line change to have privileged containers use the "docker-unconfined" profile, rather than "unconfined", and then I put this profile in /etc/apparmor.d/docker-unconfined:

#include <tunables/global>


profile docker-unconfined flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>
  network,
  capability,
  file,
  umount,
}

I don't know all the implications, but that's what finally worked for me last night.

@ewindisch

This comment has been minimized.

Show comment
Hide comment
@ewindisch

ewindisch May 27, 2015

Contributor

@bprodoehl thank you for confirming that this solves the problem. We have a series of apparmor related changes, of which this is closer to the bottom. Once we merge those, I'll officially propose this fix.

Contributor

ewindisch commented May 27, 2015

@bprodoehl thank you for confirming that this solves the problem. We have a series of apparmor related changes, of which this is closer to the bottom. Once we merge those, I'll officially propose this fix.

@bprodoehl

This comment has been minimized.

Show comment
Hide comment
@bprodoehl

bprodoehl commented May 27, 2015

@ewindisch great, thanks!

@imehrdad2012

This comment has been minimized.

Show comment
Hide comment
@imehrdad2012

imehrdad2012 Jun 23, 2015

@bprodoehl Can you please provide more details on your solution? I got stuck with the same problem, but I am not expert to immediately know how to modify similar to you.

Thanks

imehrdad2012 commented Jun 23, 2015

@bprodoehl Can you please provide more details on your solution? I got stuck with the same problem, but I am not expert to immediately know how to modify similar to you.

Thanks

@bprodoehl

This comment has been minimized.

Show comment
Hide comment
@bprodoehl

bprodoehl Jun 23, 2015

@imehrdad2012 the easiest way around it is to relocate the affected binaries in the container. Is that an option for you? Otherwise, you'll have to compile and install your own docker binary. I can post more detail on what I did, but if you can just relocate tcpdump or whatever isn't working in your container, that's a pretty easy fix.

bprodoehl commented Jun 23, 2015

@imehrdad2012 the easiest way around it is to relocate the affected binaries in the container. Is that an option for you? Otherwise, you'll have to compile and install your own docker binary. I can post more detail on what I did, but if you can just relocate tcpdump or whatever isn't working in your container, that's a pretty easy fix.

@calavera calavera closed this in #14855 Jul 23, 2015

RochesterinNYC added a commit to cloudfoundry/machete that referenced this issue Aug 4, 2016

Move tcpdump binary in docker container to workaround AppArmor issues
- This fixes the following error that running `tcpdump` in a privileged
  container generates:
  `tcpdump: error while loading shared libraries: libcrypto.so.1.0.0:
  cannot open shared object file: Permission denied`
- Problem: moby/moby#5490
- Fix: moby/moby#14855

[#126394949]

Signed-off-by: Gabriel Ramirez <gabriel.e.ramirez@gmail.com>

matt-welch added a commit to matt-welch/docker-devstack that referenced this issue Feb 2, 2017

enable docker-unconfined apparmor profile
Lots of fixes, specifically enabling docker to run with an unconstrained
profile.  Turning off apparmor seems like the easy fix, but didn't seem
to help, even when the service is disabled.
Updated .gitignore and .dockerignore to clean up docker build context.
Added script to clone openstack repositories from a list.

Idea for docker-unconfined aa_profile from:
moby/moby#5490

matt-welch added a commit to matt-welch/docker-devstack that referenced this issue Feb 2, 2017

enable docker-unconfined apparmor profile
Lots of fixes, specifically enabling docker to run with an unconstrained
profile.  Turning off apparmor seems like the easy fix, but didn't seem
to help, even when the service is disabled.
Updated .gitignore and .dockerignore to clean up docker build context.
Added script to clone openstack repositories from a list.
Edit:
Compute node seems to stack to compl;etion and virsh list --all works.
Authentication issues when invoking openstack commands.

Idea for docker-unconfined aa_profile from:
moby/moby#5490
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment