apparmor denies ptrace to docker-default profile

[stevenschlansker]

My kernel log is filling up with messages like:

[ 4473.748727] type=1400 audit(1406568919.002:236): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=12830 comm="ps" requested_mask="trace" denied_mask="trace" peer="docker-default"
[ 4478.747820] type=1400 audit(1406568923.998:237): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=12832 comm="ps" requested_mask="trace" denied_mask="trace" peer="docker-default"
[ 4478.747901] type=1400 audit(1406568923.998:238): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=12832 comm="ps" requested_mask="trace" denied_mask="trace" peer="docker-default"

docker version reports:

Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070

Ubuntu 14.04.1 3.13.0-32-generic #57-Ubuntu

[StevenVanAcker]

there's a docker-default profile in place (/etc/apparmor.d/docker) that denies ptrace. That file is recreated when docker starts and doesn't seem to be configurable(?). You can get rid of the problem by placing this profile in complain mode using aa-complain from the apparmor-utils package in Ubuntu:

root@empty64:~# aa-complain /etc/apparmor.d/docker
Setting /etc/apparmor.d/docker to complain mode.

To "fix" this more permanently, it would be great if you could get rid of apparmor alltogether:

root@empty64:~# service apparmor teardown
 * Unloading AppArmor profiles                                                                                          [ OK ] 
root@empty64:~# 

But this only works for running containers. For new containers, docker will notice there is no docker-default profile and refuse to start:

2014/07/29 07:31:52 Error response from daemon: Cannot start container 947954abf3a232ebca33f42e4bf0cec221897abb35ef3212f4cefa9c5ba73003: set apparmor profile docker-default: no such file or directory

So instead, I've now placed a line in /etc/rc.local:

aa-complain /etc/apparmor.d/docker

this survives reboots.

It would be great if docker would not recreate the apparmor profile all the time, or if the profile was somehow configurable

[crosbymichael]

I think the best way to solve issues like this is to introduce a flag so that a user can set the apparmor profile that they want for a container.

maybe something like:

--security-context apparmor:unconfined
or
--apparmor-profile unconfined

[anthony-o]

Due to that issue, I can't list files owned by a process inside a container : I have "ls: cannot read symbolic link fd/246: Permission denied" errors when I ls -lah /proc/<pid>/fd.

This question in "Unix & Linux" also talks about that.

I think issue #5936, #6800, #6607, #7330, #7147 and bug 1320869 in launchpad are more or less linked to that isn't it?

I think docker security is compromised if one puts docker AppArmor profile in complain only mode...

My question is: can we just tell AppArmor to allow ptrace for docker and is this OK to keep docker secured or is ptrace was blocked on purpose?

[crosbymichael]

@anthony-o we just introduced a --security-opts flag that lets you specifiy your own apparmor profiles. I think this will solve your issue when you need more control.

[fjorgemota]

@crosbymichael Hey. Thanks for adding --security-opts flag to Docker. :D - Can you, please, if possible, show a way to allow ptracec to be invoked in a container, solving this issue securely?

Thanks.

[jessfraz]

have you tried adding --cap-add=SYS_PTRACE?

On Thu, Oct 23, 2014 at 5:51 PM, Fernando Jorge Mota <
notifications@github.com> wrote:

@crosbymichael https://github.com/crosbymichael Hey. Thanks for adding
--security-opts flag to Docker. :D - Can you, please, if possible, show a
way to allow ptracec to be invoked in a container, solving this issue
securely?

Thanks.

—
Reply to this email directly or view it on GitHub
#7276 (comment).

[fjorgemota]

Yep.

Yet with this my /var/log/kern.log shows the following message:

    Oct 23 21:09:09 master kernel: [627511.950442] type=1400 audit(1414112949.592:28735): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=27628 comm="gdb" requested_mask="trace" denied_mask="trace" peer="docker-default"

Any other suggestion? (I need ptrace just for run gdb inside docker)

[abourget]

@fjorgemota did you fix that ? I'm having issues with gdb inside Docker too.

I got AppArmor to crash on me after an "aa-complain"

See the bug report over there: https://bugs.launchpad.net/apparmor/+bug/1390546

[fjorgemota]

For the moment I just ran aa-complain, as described in this comment

Anyway, this seems too much bad from the perspective of security, and, as is, i'm just awaiting any new suggestion about an alternative to allow gdb use from docker.

Thanks.

[abourget]

it seems --cap-add works at a different level, and is still all wrapped inside AppArmor, which doesn't let PTRACE stuff go through.

[abourget]

Another trick is to put ptrace peer=@{profile_name} in the docker apparmor profile.. but that gets overwritten upon docker daemon restarts.

You can add them to the abstraction/base profile, so it'll be used no matter how you rewrite your docker profile. This would add a cross-container security risk though, as a container could ptrace another container's processes.

[abourget]

On Ubuntu 14.04, I only managed to get ptrace through by adding ptrace, to /etc/apparmor.d/abstractions/base .. that opens cross-ptracing throughout all processes. But with that, I got rid of denials.

Linux 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[jdstrand]

I was helping abourget in #apparmor on OFTC. Here are my thoughts:

• I agree with crosbymichael that adding --security-opts is very handy. It also opens the door for per-container profiles ala libvirt-lxc, which will be good for container isolation
• for docker-default, it might also be handy to include another abstraction. Eg, "#include <abstractions/docker-local>", then have the docker daemon create an empty file (with comments) only if it doesn't already exist (thus preserving changes made by the user). This is similar to what libvirt does with libvirt-qemu and libvirt-lxc and adds flexibility
• support --enable-apparmor=false for debugging (perhaps you already do this-- I saw --enable-selinux in 1.2 but not for apparmor)
• check the parser capabilities for adding newer rules like dbus, signal, ptrace and unix. This can be done in conjunction with fixes for Update AppArmor policy based on LXC/Ubuntu policy #8454. I think the most elegant way of doing this is to do the equivalent of 'echo "profile docker-test { dbus, }" | apparmor_parser -p' for each rule type. If it returns '0', you are ok to add it. If it returns something else, skip it (or add it as a comment). The parser is smart enough to know what to do when loading policy on kernels with different capabilities, so this is a cheap and robust check

[jdstrand]

abourget's issue with needing to use a bare ptrace rule (ie, 'ptrace,') on the Ubuntu 14.04 kernel is https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1390592.

[jdstrand]

FYI, I haven't done the PR yet, but I have a branch for these updated rules and to add these rules based on the capabilities of the parser: https://github.com/jdstrand/libcontainer/tree/8454-sync-apparmor-with-lxc

[Jaykah]

Any news on this?

I would rather avoid going through AppArmor modifications (portability and all), but in some cases, like with Collectd, for example, syslog fills up so quickly, that it can really take a toll on the performance of an instance.

[Alan-R]

So, this is important (vital, really) and there is still no solution... It's really clear that docker should not be wired into requiring apparmor. What if you want to use something like SELinux?

As I read this, docker re-installs broken and untunable apparmor rules, and requires apparmor to be installed and running. If apparmor has been uninstalled, you really shouldn't try and use it. If you want to moan and complain when you start, then so be it. But you should start even if someone has opted out of apparmor.

[Alan-R]

And even when I turn the docker profile to complain, it still enforces it...

$ sudo apparmor_status
apparmor module is loaded.
1 profiles are loaded.
0 profiles are in enforce mode.
1 profiles are in complain mode.
docker-default
1 processes have profiles defined.
0 processes are in enforce mode.
1 processes are in complain mode.
docker-default (3273)
0 processes are unconfined but have a profile defined.

Process 3273 child processes (ls, readlink) are still getting permission denied when trying to look at /proc/1/exe and similar things...

[Alan-R]

Using security-opt=apparmor:unconfined makes no difference...

But adding the --privileged flag to the container when I start it seems to do the trick... Sigh...

[gerhard]

I ended up going with --privileged flag as well. Not pretty, but seemed to be the lesser of evils in my case.

[crosbymichael]

This is a combination of multiple things and it depends on what you are trying to do. It sounds like for the problems that @Alan-R and @gerhard are having you need to run with both --cap-add SYS_PTRACE and --security-opt apparmor:unconfined.

As a few examples how you could get different output:

This fails because cat is a child of sh and cannot inspect it's parents process info.

michael|~ > docker run --rm ubuntu:14.04 sh -c "cat /proc/1/environ"
cat: /proc/1/environ: Permission denied

but this works because if you are running cat as PID 1 it is allowed to inspect itself.

michael|~ > docker run --rm ubuntu:14.04 cat /proc/1/environ
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=103ad4fca658HOME=/root

With the sh -c example if I add the SYS_PTRACE cap you can see that it now works:

michael|~ > docker run --rm ubuntu:14.04 sh -c "cat /proc/1/environ"
cat: /proc/1/environ: Permission denied
michael|~ > docker run --rm --cap-add SYS_PTRACE --security-opt apparmor:unconfined ubuntu:14.04 sh -c "cat /proc/1/environ"
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=866b24a9dd35HOME=/root

Just add the cap as well as the unconfined profile on your systems and this will work as expected.

[SvenDowideit]

@crosbymichael mmm, that sounds like some useful doc eamples for caps

[Alan-R]

Pretty clearly --privileged does the same thing -- but it's not
documented exactly what that does either.

Both would be great documentation examples...

On 01/05/2015 10:17 PM, Sven Dowideit wrote:

@crosbymichael https://github.com/crosbymichael mmm, that sounds
like some useful doc eamples for caps

—
Reply to this email directly or view it on GitHub
#7276 (comment).

[stevenschlansker]

Is it possible to make this the default? Either shipped as such or via a daemon command line option? I can't possibly instruct every user of our Docker servers to add in apparmor security options with any degree of reliability, and even bare-bones basic tools like ps expect these capabilities.

[crosbymichael]

I'm not sure about the default. There could be security issues with other processes finding out information about a parent process.

I'll have to restart some of the security issues for allowing CAP_SYS_PTRACE. If anyone here has time to help research please help out and post your findings.

[jdstrand]

On 01/06/2015 03:04 PM, Steven Schlansker wrote:

Is it possible to make this the default? Either shipped as such or via a command
line option? I can't possibly instruct every user of our Docker servers to add
in apparmor security options, and even bare-bones basic tools like |ps| expect
these capabilities.

I'm in the process of creating a patch that will largely mitigate this issue by
having docker inspect the capabilities of the host to determine which rules to
apply. With this will come an updated policy for the newer rules (dbus, ptrace,
signal and unix). Attached is the preliminary patch against 1.2 if people are
interested-- I need to update to the latest docker, test under the testsuite, etc.

Jamie Strandboge http://www.ubuntu.com/

[qzio]

I believe this is fixed in experimental. See #18393
Speaking of security re ptrace I just saw http://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-8709.html I'm not sure if docker is affected by that or not though.

[jessfraz]

this is blocked by apparmor seccomp and dropping CAP_PTRACE which we do by default

On Jan 7, 2016, 05:51 -0800, joel hanssonnotifications@github.com, wrote:

I believe this is fixed in experimental. See#18393(#18393)
Speaking of security re ptrace I just sawhttp://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-8709.html(http://people.canonical.com/%7Eubuntu-security/cve/2015/CVE-2015-8709.html)I'm not sure if docker is affected by that or not though.

—
Reply to this email directly orview it on GitHub(#7276 (comment)).

[yoshiokatsuneo]

Adding both "--cap-add=SYS_PTRACE" and "--security-opt=apparmor:unconfined" does not allow container to run programs like strace/ltrace using the ptrace (PTRACE_TRACEME).

$ docker run -i -t --cap-add=SYS_PTRACE --security-opt=apparmor:unconfined --rm ubuntu sh -c 'apt-get install -y strace; strace /bin/ls'
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  strace
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 113 kB of archives.
After this operation, 504 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main strace amd64 4.8-1ubuntu5 [113 kB]
Fetched 113 kB in 1s (81.3 kB/s) 
Selecting previously unselected package strace.
(Reading database ... 11542 files and directories currently installed.)
Preparing to unpack .../strace_4.8-1ubuntu5_amd64.deb ...
Unpacking strace (4.8-1ubuntu5) ...
Setting up strace (4.8-1ubuntu5) ...
strace: test_ptrace_setoptions_for_all: PTRACE_TRACEME doesn't work: Operation not permitted
strace: test_ptrace_setoptions_for_all: unexpected exit status 1

Only "--privileded" option allow the container to use ptrace.

Is there any way to allow ptrace without using "privileged" flag ?

[jessfraz]

--security-opt seccomp:unconfined

On Tuesday, March 8, 2016, Tsuneo Yoshioka notifications@github.com wrote:

Adding both "--cap-add=SYS_PTRACE" and
"--security-opt=apparmor:unconfined" does not allow container to run
programs like strace/ltrace using the ptrace (PTRACE_TRACEME).

$ docker run -i -t --cap-add=SYS_PTRACE --security-opt=apparmor:unconfined
--rm ubuntu sh -c 'apt-get install -y strace; strace /bin/ls'
Reading package lists... Done
Building dependency tree

Reading state information... Done
The following NEW packages will be installed:
strace
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 113 kB of archives.
After this operation, 504 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main strace amd64
4.8-1ubuntu5 [113 kB]
Fetched 113 kB in 1s (81.3 kB/s)
Selecting previously unselected package strace.
(Reading database ... 11542 files and directories currently installed.)
Preparing to unpack .../strace_4.8-1ubuntu5_amd64.deb ...
Unpacking strace (4.8-1ubuntu5) ...
Setting up strace (4.8-1ubuntu5) ...
strace: test_ptrace_setoptions_for_all: PTRACE_TRACEME doesn't work:
Operation not permitted
strace: test_ptrace_setoptions_for_all: unexpected exit status 1

Only "--privileded" option allow the container to use ptrace.

Is there any way to allow ptrace without using "privileged" flag ?

—
Reply to this email directly or view it on GitHub
#7276 (comment).

Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3
pgp.mit.edu http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3

[yoshiokatsuneo]

@jfrazelle Ah, it worked ! Thanks ! And, if possible, can you explain the difference between sec comp:unconfined and apparmor:unconfined ?

[jessfraz]

I would checkout the security docs for a detailed description apparmor is a
LSM that is defined by a profile, seccomp allows defining behavior for
specific syscalls wrt allow or error

On Tuesday, March 8, 2016, Tsuneo Yoshioka notifications@github.com wrote:

@jfrazelle https://github.com/jfrazelle Ah, it worked ! Thanks ! And,
if possible, can you explain the difference between sec comp:unconfined and
apparmor:unconfined ?

—
Reply to this email directly or view it on GitHub
#7276 (comment).

Jessie Frazelle
4096R / D4C4 DD60 0D66 F65A 8EFC 511E 18F3 685C 0022 BFF3
pgp.mit.edu http://pgp.mit.edu/pks/lookup?op=get&search=0x18F3685C0022BFF3

[yoshiokatsuneo]

@jfrazelle Ah, so, in my understanding, "--security-opt seccomp:XXX" and "--security-opt apparmor:XXX" is independent option.
"--security-opt apparmor:unconfined" only disable apparmor and does not disable seccomp.
"--security-opt seccomp:unconfined" only disable seccomp and does not disable apparmor.

To disable both, both "--security-opt apparmor:unconfined" and "--security-opt seccomp:unconfined" need to specified.
I think I got it. Thanks !

[gswallow]

--security-opt seccomp:unconfined doesn't work for me. I'm Ubuntu 14.04.

I just used apparmor:unconfined, and I'm not getting errors in dmesg. Is this a "newer" docker thing, or a "newer" ubuntu thing?

[yoshiokatsuneo]

@gswallow
For me, following command works on both (Ubuntu 14.0.4, kernel 4.2.0-30-generic, docker1.10.3), and on (Ubuntu 15.10, kernel 4.2.0-18, docker1.10.2, provisioned by docker-machine).

  $ docker run -i -t --security-opt=seccomp:unconfined --rm ubuntu sh -c 'apt-get install -y strace; strace /bin/ls'

[gswallow]

It's a docker 1.10 thing. --security-opt=seccomp:unconfined is covered in the changelog.

We're using Amazon ECS, which says to use Docker 1.9.1. From what I understand, unless docker calls PR_SET_SECCOMP, then it doesn't matter.