Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing to stop containers, if apparmor enabled. #36809

Closed
mrueg opened this issue Apr 6, 2018 · 10 comments

Comments

@mrueg
Copy link
Contributor

commented Apr 6, 2018

Description
We tested docker-ce 17.12.1 and docker-ce 18.03.0 with apparmor enabled, and docker fails to stop the actual containers.

Steps to reproduce the issue:

  1. docker run gcr.io/google_containers/pause-amd64:3.0
  2. docker stop $container_id_of_step1

Describe the results you received:

docker stop $containerid
Error response from daemon: cannot stop container: $containerid: Cannot kill container $containerid: unknown error after kill: docker-runc did not terminate sucessfully: container_linux.go:393: signaling init process caused "permission denied"
: unknown

dmesg output
[ 551.981297] audit: type=1400 audit(1523033503.323:9): apparmor="DENIED" operation="signal" profile="docker-default" pid=7162 comm="docker-runc" requested_mask="receive" denied_mask="receive" signal=stp peer="unconfined"

Describe the results you expected:
container should have been stopped

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

docker version
Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.10.1
 Git commit:    0520e24
 Built: Fri Apr  6 16:37:32 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.10.1
  Git commit:   0520e24
  Built:        Fri Apr  6 16:35:44 2018
  OS/Arch:      linux/amd64
  Experimental: true

Output of docker info:

docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 5
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd0439 (expected: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c)
runc version: 4fc53a8 (expected: 4fc53a81fb7c994640722ac585fa9ca548971871)
init version: v0.17.0 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.14.32-grsecurity-r1-docker-2
Operating System: Gentoo/Linux
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.841GiB
Name: zip
ID: zip
Docker Root Dir: zip
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
@cyphar

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2018

We've seen this recently as well. It's caused by an kernel-side apparmor change, and requires adding the following hunk. I will submit a PR which includes it.

From: Goldwyn Rodrigues <rgoldwyn@suse.com>
Subject: Allow signal mediation while for apparmor profile

Allows docker processes under docker-default ot receive all signals.

Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
---
 components/engine/profiles/apparmor/template.go |    1 +
 1 file changed, 1 insertion(+)

--- a/components/engine/profiles/apparmor/template.go
+++ b/components/engine/profiles/apparmor/template.go
@@ -17,6 +17,7 @@ profile {{.Name}} flags=(attach_disconne
   capability,
   file,
   umount,
+  signal (receive) peer=unconfined,
 
   deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
   # deny write to files not in /proc/<number>/** or /proc/sys/**

gentoo-bot pushed a commit to gentoo/gentoo that referenced this issue Apr 6, 2018

app-emulation/docker: Add Patch
moby/moby#36809

Package-Manager: Portage-2.3.28, Repoman-2.3.9
@vniche

This comment has been minimized.

Copy link

commented May 23, 2018

For the ones that intend to solve this issue immediately, i was suffering from the same symptom, after some searching i've found this response which led me to remove snap apparmor profiles, remaining settings from a previous installation:
sudo rm -rf /var/lib/snapd/apparmor/profiles/snap.docker.*
Warning: recommended only for non-snap installations.

isi-lincoln added a commit to ceftb/prototypes that referenced this issue Jul 9, 2018

docker-compose: fix apparmor on docker hosts
Due to an issue in apparmor/docker integration:
moby/moby#36809

It is necessary to pass:

security_opt:
  - seccomp:unconfined

to each of the docker-compose images.

https://docs.docker.com/engine/security/seccomp/
"You can pass unconfined to run a container without the
default seccomp profile."

My understanding is this disables the docker default whitelist
to allow more dangerous kernel syscalls which may allow for easier
jailbreaking.
@ericchiang

This comment has been minimized.

Copy link

commented Sep 11, 2018

I just hit this on a 4.17 kernel.

The original kernel change was reverted in 4.14, but apparently was re-targeted for a later release #36822 (comment). Don't know if it got in after 4.16, which was confirmed as unaffected.

@cyphar

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2018

We appear to be carrying this patch in SLE, but if it's being carried in other kernels then I could re-open #36822.

@matthewreimer

This comment has been minimized.

Copy link

commented Sep 11, 2018

I'm seeing this same problem on plain vanilla Ubuntu Server 18.04.1 installed yesterday, with the following versions:

kernel 4.15.0-34-generic
apparmor 2.12-4ubuntu5
docker-ce 18.06.1ce3-0~ubuntu

Is there a workaround until this gets fixed upstream?

@cyphar

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2018

You'd need to either disable AppArmor (--security-opt apparmor=unconfined but this reduces security) or replace the AppArmor profile (note that this isn't a permanent fix, Docker will automatically re-load the broken docker-default profile if it ever gets unloaded):

% cat replaced-profile
#include <tunables/global>

profile docker-default flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>

  network,
  capability,
  file,
  umount,

  # Allow signals from privileged profiles, and from within the same profile.
  signal (receive) peer=unconfined,
  signal (send,receiver) peer=docker-default,

  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,
  deny mount,
  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/kernel/security/** rwklx,

  # suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
  ptrace (trace,read) peer={{.Name}},
}
% sudo apparmor_parser -r replaced-profile
@mrueg

This comment has been minimized.

Copy link
Contributor Author

commented Sep 12, 2018

@cyphar if you could reopen the PR with signal (send,receiver) peer=docker-default, included, that would be perfect. We came across this issue with a java container which application used realtime signals, that got killed by apparmor.

@cyphar

This comment has been minimized.

Copy link
Contributor

commented Nov 14, 2018

(#37831 is the PR.)

@CampGareth

This comment has been minimized.

Copy link

commented Dec 11, 2018

@cyphar

I'm attempting to replace the Apparmor profile as a workaround until Docker 18.09.1 is available but get the following error when running sudo apparmor_parser -r replaced-profile, any suggestions?

AppArmor parser error for replaced-profile in replaced-profile at line 13: Internal: unexpected signal mode character 'e' in input

I'm running Apparmor version 2.12-4ubuntu5.1 on Ubuntu 18.04.1 LTS

edit

After playing around with the profile and diffing it against the original at /etc/apparmor.d/docker I ended up with the following which works:

#include <tunables/global>

profile docker-default flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>

  network,
  capability,
  file,
  umount,

  # Allow signals from privileged profiles, and from within the same profile.
  signal (receive) peer=unconfined,
  signal (send,receive) peer=docker-default,

  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,
  deny mount,
  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/kernel/security/** rwklx,

  # suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
  ptrace (trace,read) peer=docker-default,

}
@hellojukay

This comment has been minimized.

Copy link

commented Jan 2, 2019

i have th same problem on ubuntu 18.10

infra@huangchengkai-FFFFFFF:/data/loki$ docker rm -f loki-server
Error response from daemon: Could not kill running container 455980de7979471a2d1f869068f89854f13a6e6d7f04865a4cb87bc66cef3e0b, cannot remove - Cannot kill container 455980de7979471a2d1f869068f89854f13a6e6d7f04865a4cb87bc66cef3e0b: unknown error after kill: runc did not terminate sucessfully: container_linux.go:378: signaling init process caused "permission denied"
: unknown
infra@huangchengkai-FFFFFFF:/data/loki$ ps aux | grep loki
infra    21400  0.0  0.0 129344 22912 ?        Ssl  17:27   0:01 /bin/loki -config.file=/etc/loki/local-config.yaml
infra    23624  0.0  0.0  17480   892 pts/1    S+   17:50   0:00 grep loki
infra@huangchengkai-FFFFFFF:/data/loki$ sudo kill -9 21400
kill: (21400): Permission denied
infra@huangchengkai-FFFFFFF:/data/loki$
infra@huangchengkai-FFFFFFF:/data/loki$ uname -a
Linux huangchengkai-FFFFFFF 4.18.0-13-generic #14-Ubuntu SMP Wed Dec 5 09:04:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
infra@huangchengkai-FFFFFFF:/data/loki$

@wei wei referenced this issue Feb 18, 2019

Closed

无法删除镜像 #33

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.