libcontainer: workaround for empty key label bug in RHEL7 #2070

kolyshkin · 2019-06-10T22:55:21Z

On a freshly installed CentOS 7 system (kernel 3.10.0-957.12.2.el7.x86_64)
with SELinux enabled, the kernel does not allow to write an empty label
to the keycreate file. Here's an strace except from containerd:

20396 openat(AT_FDCWD, "/proc/self/attr/keycreate", O_WRONLY|O_CLOEXEC) = 7
...
20396 write(7, "", 0) = -1 EACCES (Permission denied)

NOTE: upgrading or downgrading container-selinux makes this to go
away, so in order to reproduce you need a freshly booted system.

Here is a reproducer in C:

[root@kir-ce7-selinux-01 ~]# cat a.c
 #include <sys/types.h>
 #include <sys/stat.h>
 #include <fcntl.h>
 #include <unistd.h>
 #include <stdio.h>

int main(void) {
    int fd, r;

    fd = open("/proc/self/attr/keycreate", O_WRONLY);
    if (fd < 0) {
	    perror("open");
    }

    r = write(fd, "", 0);
    if (r < 0) {
	    perror("write");
    }

    return 0;
}

[root@kir-ce7-selinux-01 ~]# uname -a
Linux kir-ce7-selinux-01 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@kir-ce7-selinux-01 ~]# gcc -Wall -O2 -o a a.c

[root@kir-ce7-selinux-01 ~]# ./a
write: Permission denied
[root@kir-ce7-selinux-01 ~]# strace ./a
...
open("/proc/self/attr/keycreate", O_WRONLY) = 3
write(3, "", 0)                         = -1 EACCES (Permission denied)
...

So, this is presumably a kernel bug, but since there is no update
available for this kernel, we have to have this workaround here.

This fixes non-working docker/containerd/runc on CentOS 7 (and,
presumably, RHEL 7, too).

Signed-off-by: Kir Kolyshkin kolyshkin@gmail.com

On a freshly installed CentOS 7 system (kernel 3.10.0-957.12.2.el7.x86_64) with SELinux enabled, the kernel does not allow to write an empty label to the keycreate file. Here's an strace except from containerd: > 20396 openat(AT_FDCWD, "/proc/self/attr/keycreate", O_WRONLY|O_CLOEXEC) = 7 > ... > 20396 write(7, "", 0) = -1 EACCES (Permission denied) NOTE: upgrading or downgrading container-selinux makes this to go away, so in order to reproduce you need a freshly booted system. Here is a reproducer in C: ``` [root@kir-ce7-selinux-01 ~]# cat a.c #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <unistd.h> #include <stdio.h> int main(void) { int fd, r; fd = open("/proc/self/attr/keycreate", O_WRONLY); if (fd < 0) { perror("open"); } r = write(fd, "", 0); if (r < 0) { perror("write"); } return 0; } [root@kir-ce7-selinux-01 ~]# uname -a Linux kir-ce7-selinux-01 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux [root@kir-ce7-selinux-01 ~]# gcc -Wall -O2 -o a a.c [root@kir-ce7-selinux-01 ~]# ./a write: Permission denied [root@kir-ce7-selinux-01 ~]# strace ./a ... open("/proc/self/attr/keycreate", O_WRONLY) = 3 write(3, "", 0) = -1 EACCES (Permission denied) ... ``` So, this is presumably a kernel bug, but since there is no update available for this kernel, we have to have this workaround here. This fixes non-working docker/containerd/runc on CentOS 7 (and, presumably, RHEL 7, too). Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin · 2019-06-10T22:55:54Z

This is an alternative to #2033 in case opencontainers/selinux#56 won't be accepted.

cyphar · 2019-06-10T23:06:38Z

/cc @rhatdan

cyphar · 2019-06-10T23:07:34Z

libcontainer/setns_init_linux.go

@@ -35,7 +35,10 @@ func (l *linuxSetnsInit) Init() error {

 	if !l.config.Config.NoNewKeyring {
 		if err := label.SetKeyLabel(l.config.ProcessLabel); err != nil {
-			return err
+			// workaround for RHEL7 kernel which may return EACCES for an empty label
+			if !(l.config.ProcessLabel == "" && os.IsPermission(err)) {


I think if l.config.ProcessLabel != "" || !os.IsPermission(err) reads nicer.

To me your version is less readable (despite looking better and being equally correct). I read mine as "ignore the error unless label is empty and error is permission".

@cyphar but I can certainly change it, please let me know

I don't really mind either way. I read mine as "this is an error if the label is not empty, or the error is not a permission-related one" but either works.

kolyshkin · 2019-06-11T02:38:24Z

Just checked that this helps to avoid the issue. Still see the AVC but it no longer affects the operation.

[root@kir-centos-selinux-3 ~]# cp /usr/sbin/runc /usr/sbin/runc-orig
[root@kir-centos-selinux-3 ~]# cp /usr/sbin/runc-my /usr/sbin/runc
cp: overwrite ‘/usr/sbin/runc’? y

[root@kir-centos-selinux-3 ~]# docker run hello-world

Hello from Docker!
....

[root@kir-centos-selinux-3 ~]# runc-orig --version
runc version 1.0.0-rc8
commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
spec: 1.0.1-dev

[root@kir-centos-selinux-3 ~]# runc-my --version
runc version 1.0.0-rc8+dev
commit: 1509446a7ff844302641c9fa871ddcc3eac6d38e
spec: 1.0.1-dev

[root@kir-centos-selinux-3 ~]# ausearch -m avc -ts recent
----
time->Tue Jun 11 02:35:25 2019
type=PROCTITLE msg=audit(1560220525.865:173): proctitle=72756E6300696E6974
type=SYSCALL msg=audit(1560220525.865:173): arch=c000003e syscall=1 success=no exit=-13 a0=7 a1=557e9544f158 a2=0 a3=0 items=0 ppid=11486 pid=11495 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc:[2:INIT]" exe="/" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=AVC msg=audit(1560220525.865:173): avc:  denied  { create } for  pid=11495 comm="runc:[2:INIT]" scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=key permissive=0
----
time->Tue Jun 11 02:35:52 2019
type=PROCTITLE msg=audit(1560220552.918:177): proctitle=72756E6300696E6974
type=SYSCALL msg=audit(1560220552.918:177): arch=c000003e syscall=1 success=no exit=-13 a0=6 a1=55855645e638 a2=0 a3=0 items=0 ppid=11538 pid=11547 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runc:[2:INIT]" exe="/" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=AVC msg=audit(1560220552.918:177): avc:  denied  { create } for  pid=11547 comm="runc:[2:INIT]" scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=key permissive=0

crosbymichael · 2019-06-11T11:57:59Z

LGTM

crosbymichael · 2019-06-11T12:00:22Z

ping @cyphar can you review this one again. I looked into this and I think its good to handle this at the runc level and let the selinux pkg just pass through errors.

crosbymichael · 2019-06-11T12:06:27Z

cc @opencontainers/runc-maintainers

thaJeztah

LGTM (not a maintainer)

crosbymichael · 2019-06-11T21:16:40Z

ping @mrunalp @cyphar

rhatdan · 2019-06-11T21:33:32Z

Again you are ignoring a legitimate denial of permissions, that could become serious in the future. The issue is to fix the SElinux policy not to ignore Permission Denied.
A process is attempting to disable the default labelling. If you update container-selinux then the denial goes away and the process is able to reset the SELinux label to the default settings.

rhatdan · 2019-06-11T21:36:53Z

If you want to allow this for now, you can simply

# grep container_t /var/log/audit/audit.log | audit2allow -M mycontainer
# semodule -i mycontainer.pp

This will allow the domain to set the key label.

Is runc running in an init script or is it started by Docker/Containerd. If the later then Docker/Containerd is labeled incorrectly.

ls -lZ /usr/bin/docker* /usr/bin/containerd* /usr/bin/runc*

crosbymichael · 2019-06-11T22:10:23Z

@rhatdan on package install, should restorecon be called in the rpc scripts post install?

kolyshkin · 2019-06-11T23:00:44Z

OK the reason for this was that

our containerd.io did not require container-selinux. Easy to fix.
container-selinux rpm %post script calls restorecon ... /usr/bin/runc while we install it to /usr/sbin and so it is not labeled after container-selinux is installed.

rhatdan · 2019-06-12T08:11:21Z

It is better to do a Recommends then a Requires, since some people might install contained.io inside of a container, and you probably don't want to pull selinux-policy into a container.

kolyshkin · 2019-06-12T19:45:22Z

It is better to do a Recommends then a Requires, since some people might install contained.io inside of a container, and you probably don't want to pull selinux-policy into a container.

Alas, RHEL7 has rpm-4.11 which does not understand Recommends: (which appeared in rpm-4.12 and allegedly not backported to RHEL7).

rhatdan · 2019-06-12T21:44:11Z

Sorry I did not realize this was for RHEL7, you are correct.

rhatdan · 2019-06-12T21:57:37Z

BTW @kolyshkin The SELinux labeling issue with KeyCreate turns out to be a thirteen year old bug in the kernel. Which just got merged into the SELinux kernel, and should be moving it's way into the upstream kernel, and eventually possibly back into RHEL7.

cyphar reviewed Jun 10, 2019

View reviewed changes

kolyshkin mentioned this pull request Jun 11, 2019

go-selinux/SetKeyLabel: fix for RHEL7 kernels opencontainers/selinux#56

Merged

thaJeztah approved these changes Jun 11, 2019

View reviewed changes

This was referenced Jun 11, 2019

[19.03 backport] docker-ce.spec: bump container-selinux req docker/docker-ce-packaging#340

Closed

docker-ce.spec: bump container-selinux req docker/docker-ce-packaging#339

Closed

crosbymichael closed this Jun 11, 2019

kolyshkin deleted the empty-key branch April 21, 2020 08:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

libcontainer: workaround for empty key label bug in RHEL7 #2070

libcontainer: workaround for empty key label bug in RHEL7 #2070

kolyshkin commented Jun 10, 2019

kolyshkin commented Jun 10, 2019

cyphar commented Jun 10, 2019

cyphar Jun 10, 2019

kolyshkin Jun 11, 2019 •

edited

kolyshkin Jun 11, 2019

cyphar Jun 11, 2019

kolyshkin commented Jun 11, 2019

crosbymichael commented Jun 11, 2019 •

edited by caniszczyk

crosbymichael commented Jun 11, 2019

crosbymichael commented Jun 11, 2019

thaJeztah left a comment

crosbymichael commented Jun 11, 2019

rhatdan commented Jun 11, 2019

rhatdan commented Jun 11, 2019

crosbymichael commented Jun 11, 2019

kolyshkin commented Jun 11, 2019

rhatdan commented Jun 12, 2019

kolyshkin commented Jun 12, 2019

rhatdan commented Jun 12, 2019

rhatdan commented Jun 12, 2019

libcontainer: workaround for empty key label bug in RHEL7 #2070

libcontainer: workaround for empty key label bug in RHEL7 #2070

Conversation

kolyshkin commented Jun 10, 2019

kolyshkin commented Jun 10, 2019

cyphar commented Jun 10, 2019

cyphar Jun 10, 2019

Choose a reason for hiding this comment

kolyshkin Jun 11, 2019 • edited

Choose a reason for hiding this comment

kolyshkin Jun 11, 2019

Choose a reason for hiding this comment

cyphar Jun 11, 2019

Choose a reason for hiding this comment

kolyshkin commented Jun 11, 2019

crosbymichael commented Jun 11, 2019 • edited by caniszczyk

crosbymichael commented Jun 11, 2019

crosbymichael commented Jun 11, 2019

thaJeztah left a comment

Choose a reason for hiding this comment

crosbymichael commented Jun 11, 2019

rhatdan commented Jun 11, 2019

rhatdan commented Jun 11, 2019

crosbymichael commented Jun 11, 2019

kolyshkin commented Jun 11, 2019

rhatdan commented Jun 12, 2019

kolyshkin commented Jun 12, 2019

rhatdan commented Jun 12, 2019

rhatdan commented Jun 12, 2019

kolyshkin Jun 11, 2019 •

edited

crosbymichael commented Jun 11, 2019 •

edited by caniszczyk