MQ does not run on containerd

[rogeriao]

We upgrade our kubernetes cluster on IBM Cloud to version 1.11.2_1516 and the mq docker image stopped working with the below error:

2018-09-06T05:53:41.778Z Set password for “admin” user
2018-09-06T05:53:41.795Z Set password for “app” user
2018-09-06T05:53:41.803Z Using queue manager name: QM1
2018-09-06T05:53:41.803Z CPU architecture: amd64
2018-09-06T05:53:41.803Z Linux kernel version: 4.4.0-133-generic
2018-09-06T05:53:41.803Z Container runtime: kube
2018-09-06T05:53:41.803Z Base image: Ubuntu 16.04.5 LTS
2018-09-06T05:53:41.804Z Maximum file handles: 6000000
2018-09-06T05:53:41.804Z Running as user ID 0 (root) with primary group 0, and supplemental groups 999
2018-09-06T05:53:41.804Z Capabilities (bounding set): chown,dac_override,fowner,fsetid,kill,setgid,setuid,setpcap,net_bind_service,net_raw,sys_chroot,mknod,audit_write,setfcap
2018-09-06T05:53:41.804Z seccomp enforcing mode: disabled
2018-09-06T05:53:41.804Z Process security attributes: cri-containerd.apparmor.d (enforce)
2018-09-06T05:53:41.805Z No volume detected. Persistent messages may be lost
2018-09-06T05:53:41.815Z Created directory structure under /var/mqm
2018-09-06T05:53:41.815Z Image created: 2018-08-14T14:06:55+00:00
2018-09-06T05:53:41.816Z Image revision: 4ca342251d1a7f2afc4f85fc1831316687dcdca9
2018-09-06T05:53:41.816Z Image source: https://github.com/ibm-messaging/mq-container.git
2018-09-06T05:53:41.831Z MQ version: 9.1.0.0
2018-09-06T05:53:41.831Z MQ level: p910-L180709.DE
2018-09-06T05:53:41.831Z MQ license: Developer
2018-09-06T05:53:41.832Z Creating queue manager QM1
2018-09-06T05:53:41.832Z Starting web server
2018-09-06T05:53:41.840Z crtmqm returned 119
2018-09-06T05:53:41.840Z AMQ7077E: You are not authorized to perform the requested operation.

Thanks

[rogeriao]

Hi guys, just to give more information on the things we did:

• We tried with other MQ versions and still got the issue
• We tried to use a persistent volume

Thanks

[estesp]

This is a deviation in default behavior in docker vs. containerd with respect to user IDs and supplemental groups. Usually when running as root, containerd doesn't parse /etc/group to see if uid 0/root has been placed in supplemental groups; therefore they are not added to the OCI spec.

Docker always looks up the user in the /etc/group of the container root filesystem and adds any supplemental groups to the OCI default spec when running the container.

This means in the MQ case that root being in group mqm is ignored via the containerd/CRI path, causing your commands which are setgid mqm to fail. I will be creating PRs to have the CRI plugin (for Kubernetes) when running containerd to add the supplemental groups to match Docker behavior.

[estesp]

Can anyone provide me a way to run crtmqm separately from the default flow of the default container startup program? I can shell into the ibmcom/mq container, but running /opt/mqm/bin/crtmqm says it doesn't know the queue name or something, and won't actually attempt whatever step fails permissions. Still making sure we have all the right pieces to enable this container to work when containerd is the runtime. @rogeriao

[adasilva70]

@estesp the command that is failing is crtmqm -q -p 1414 QM1 - is that what you need?

[estesp]

I have successfully tested running the mq:9.1 container under containerd with 2 changes required:

• the additional GIDs PR that is ready for merging in containerd (Add With-helper for supplemental gid support containerd/containerd#2627), and for which the containerd CRI plugin will need to be updated to use for default containers running under Kubernetes (Fix supplemental group id containerd/cri#910)
• turning off the No New Privileges flag in the OCI default spec in containerd. This is trickier and will need discussion upstream. Docker has it off by default, but can be turned on with --security-opt=no-new-privileges on the docker run command line. Containerd has it on by default, and there are no current knobs to turn it off, without creating your own spec modification when calling containerd.

Item 1 is well underway; item 2 will need some discussion to see how we can solve. In your case, the no_new_privs flag on the container prevents root from "switching" to the mqm effective gid via the setgid bit on many of the MQ binaries.

[Random-Liu]

@estesp Actually, I found that in cri plugin, we apply NoNewPrivileges purely based on CRI input.

It should overwrite the containerd default. https://github.com/containerd/cri/blob/master/pkg/server/container_create.go#L411

It seems that in IBM cloud, the default value of AllowPrivilegeEscalation is set to false?

[estesp]

I should have investigated further; I did not realize CRI was already setting no_new_privs to false and I was “cheating” by testing with my changes just using ctr. The reason it didn’t work in IBM Cloud is that without the helper to get the additional GIDs set in the spec, it didn’t get far enough to “test” no_new_privs, but because containerd was setting it to true I was immediately hitting that issue after fixing the additional GIDs problem. Once CRI uses the new helper for addl GIDs, we can test via CRI and should work now. A separate issue is probably to make a way to easily test a container with CRI-like spec config via ctr (or just use crictl I guess?)

[arthurbarr]

This has been fixed by a change to containerd.