selinux keycreate issue

[perezjasonr]

seems to be an issue stemming from container-selinux is occurring in rhel8 if installing containerd. Ive noticed in Rhel7 folks say to update container-selinux.

docker/for-linux#983

moby/moby#39109

I'm getting it in Rhel8, my guess is that the container-selinux that comes with containerd might be one that doesn't have the fix mentioned. I think this is noteworthy because many will attempt to follow these instructions for their containerd installation:

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

I guess it depends on what exactly the container is trying to do because not all of them get it.

Steps to reproduce the issue:

1. install containerd with yum/dnf
2. ensure selinux is enforcing, and that its enabled in the containerd conf
3. run a container that may attempt to interact with keycreate, in my case its gitlab's "certificates" container: alpine-certificates:20191127-r2

Describe the results you received:

some containers get this error:

Error: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: failed to set /proc/self/attr/keycreate on procfs: write /proc/self/attr/keycreate: invalid argument: unknown

Describe the results you expected:

should start up like the other containers

What version of containerd are you using:

containerd.io.x86_64               1.4.9-3.1.el8

containerd --version
containerd containerd.io 1.4.6 d71fcd7d8303cbf684402823e425e9dd2e99285d

Any other relevant information (runC version, CRI configuration, OS/Kernel version, etc.):

runc --version

runc --version
runc version 1.0.0-rc95
commit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
spec: 1.0.2-dev
go: go1.13.15
libseccomp: 2.5.1

selinux enabled/true

crictl info

$ crictl info

uname -a

$ uname -a

uname -a
Linux ip-10-42-32-9.ec2.internal 4.18.0-305.12.1.el8_4.x86_64 #1 SMP Mon Jul 26 08:06:24 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux

probably also useful:

dnf list --installed | grep container-selinux
container-selinux.noarch                      2:2.164.1-1.module+el8.4.0+11870+8b6f7018   @rhel-8-appstream-rhui-rpms

and dnf does not seem to think theres an upgrade for this available for a newer version. In the links above they are talking about versions like 2.9

[thaJeztah]

It looks like you're running an older version of containerd and runc (containerd.io 1.4.6 and runc 1.0.0-rc95). Are you seeing the same on the latest containerd.io package (1.4.9), which comes with runc 1.0.1 ?

[perezjasonr]

I suppose that is worth a try, i didnt know about 1.4.9, as 1.4.6 was newest last I checked.
Now that said, this also means 1.4.6 is still, quite new:

2021-05-21 - Sebastiaan van Stijn <thajeztah@docker.com> - 1.4.6-3.1
- Update to containerd 1.4.6
- Update runc to v1.0.0-rc95 to address CVE-2021-30465.
2021-05-12 - Sebastiaan van Stijn <thajeztah@docker.com> - 1.4.5-3.1
- Update to containerd 1.4.5
- Update runc to v1.0.0-rc94

also I noticed they have the same value here:

container-selinux | >= 2.74

[perezjasonr]

trying it right now and will update with results.

[thaJeztah]

Thanks! I definitely don't know if it will resolve the issue, but just in case (I know there were some fixes in runc 1.0.1 around SELinux handling)

[perezjasonr]

unfortunately I'm still seeing it:

Warning Failed 2m26s (x4 over 3m53s) kubelet Error: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: failed to set /proc/self/attr/keycreate on procfs: write /proc/self/attr/keycreate: invalid argument: unknown

although actually it grabbed crun? heres the relevant versions:

dnf list --installed | grep containerd
containerd.io.x86_64                         1.4.9-3.1.el8                               @docker-ce-stable
dnf list --installed | grep container-selinux
container-selinux.noarch                      2:2.164.1-1.module+el8.4.0+11870+8b6f7018   @rhel-8-appstream-rhui-rpms
[root@ip-10-42-20-190 ~]# dnf list --installed | grep runc
      [ no output ]
[root@ip-10-42-20-190 ~]# dnf list --installed | grep run
crun.x86_64                                   0.20.1-1.module+el8.4.0+11822+6cc1e7d7      @rhel-8-appstream-rhui-rpms

edit:

wait actually runc is there:

runc --version
runc version 1.0.1
commit: v1.0.1-0-g4144b63
spec: 1.0.2-dev
go: go1.15.14
libseccomp: 2.5.1

i guess it just wasn't a package listed in dnf,

this is unfortunate as, for some of us disabling selinux isn't an option. and of the bigger updates in containerd 1.4 was better selinux support (like for actual contexts/labeling, and such)

[perezjasonr]

I may have figured out something, let me try to verify it with some trial and error likely ill have something by tomorrow morning.

[perezjasonr]

Ok so I think I may have figured this out and can probably close the issue. I don't think the ball is in containerd's court in this case unless they want to handle this exception and possibly present it differently (if thats even possible).

But I think its worth explaining for future users who run into this.

The error here appears to be misleading, because it LOOKS like the same from the issues I linked in the original post. However,
I think the final part:

invalid argument: unknown

might be key. I noticed that which containers got this was changing, namely, increasing, and then decreasing (after I became suspicious in my last post). The correlation was providing an selinux context that doesn't exist. I had removed the policy but forgot to remove the pod manifest entry instructing it to use the selinux context.

in other words you get this if you try to give a container/pod selinuxOptions type that doesn't exist.
needless to say, after cleaning that up, I don't see the error anymore.

let me know if you think we can close this issue, or if containerd is interested in it in some way shape or form, but doesn't seem to be a containerd issue to me.

[thaJeztah]

Good to hear you found the cause!

As to the error message; all errors after OCI Runtime create failed: are coming from the OCI runtime (runc in this case), so this part is what containerd received from runc;

container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: failed to set /proc/self/attr/keycreate on procfs: write /proc/self/attr/keycreate: invalid argument: unknown

I agree those errors are quite verbose, but not really helpful either. Not sure if runc would have more information available to produce a better error, but if you have an easy way to reproduce, it might be worth opening a ticket in the runc repository; https://github.com/opencontainers/runc

[thaJeztah]

Let me close this ticket per the above, but feel free to continue the conversation