Privileged containers fail on CentOS

[pires]

Can't run privileged containers with the following set-up:

• CentOS 7.x, 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Kubernetes 1.9.0
• containerd 1.0.0
• cri-containerd v1.0.0-beta.0

How to reproduce:

$ cat <<EOF | kubectl create -f -
kind: Pod
apiVersion: v1
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    securityContext:
      privileged: true
    image: nginx:1.12.2-alpine
EOF

What we're seeing:

$ kubectl get po
NAME      READY     STATUS              RESTARTS   AGE
nginx     0/1       ContainerCreating   0          4s

$ kubectl describe po nginx
Name:         nginx
Namespace:    default
Node:         ip-10-10-0-246.us-west-2.compute.internal/10.10.0.246
Start Time:   Wed, 03 Jan 2018 15:22:34 +0000
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           9.0.3.5
Containers:
  nginx:
    Container ID:   cri-containerd://5337661abdf7b4e538dc3f89d3461af9aeb1203e70939982847f11672bc642b3
    Image:          nginx:1.12.2-alpine
    Image ID:       docker.io/library/nginx@sha256:36b0181554913b471ae33546a9c19cc80e97f44ce5e7234995e307f14da57268
    Port:           <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: OCI runtime create failed: container_linux.go:296: starting container process caused "unknown capability \"CAP_AUDIT_READ\"": unknown
      Exit Code:    128
      Started:      Thu, 01 Jan 1970 00:00:00 +0000
      Finished:     Wed, 03 Jan 2018 15:22:40 +0000
    Ready:          False
    Restart Count:  1
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vtdsp (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  default-token-vtdsp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-vtdsp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age              From                                                Message
  ----     ------                 ----             ----                                                -------
  Normal   Scheduled              10s              default-scheduler                                   Successfully assigned nginx to ip-10-10-0-246.us-west-2.compute.internal
  Normal   SuccessfulMountVolume  10s              kubelet, ip-10-10-0-246.us-west-2.compute.internal  MountVolume.SetUp succeeded for volume "default-token-vtdsp"
  Normal   Pulling                8s               kubelet, ip-10-10-0-246.us-west-2.compute.internal  pulling image "nginx:1.12.2-alpine"
  Normal   Pulled                 5s               kubelet, ip-10-10-0-246.us-west-2.compute.internal  Successfully pulled image "nginx:1.12.2-alpine"
  Normal   Created                5s (x2 over 5s)  kubelet, ip-10-10-0-246.us-west-2.compute.internal  Created container
  Normal   Pulled                 5s               kubelet, ip-10-10-0-246.us-west-2.compute.internal  Container image "nginx:1.12.2-alpine" already present on machine
  Warning  Failed                 4s (x2 over 5s)  kubelet, ip-10-10-0-246.us-west-2.compute.internal  Error: failed to create containerd task: OCI runtime create failed: container_linux.go:296: starting container process caused "unknown capability \"CAP_AUDIT_READ\"": unknown
  Warning  BackOff                2s (x2 over 4s)  kubelet, ip-10-10-0-246.us-west-2.compute.internal  Back-off restarting failed container

However, Docker CE 17.12, which runs the same containerd version, works fine.

cc @bmcstdio

[Random-Liu]

@pires Thanks for reporting!
For kernel older than CAP_AUDIT_READ is not supported. http://man7.org/linux/man-pages/man7/capabilities.7.html

       CAP_AUDIT_READ (since Linux 3.16)
              Allow reading the audit log via a multicast netlink socket.

We need to enable the host specific logic in runtime-tools generator https://github.com/opencontainers/runtime-tools/blob/master/generate/generate.go#L31. I'll send out a PR soon.

[pires]

Thanks @Random-Liu you just saved us a bunch of time - we were looking into a fix too but you got there really fast.