Issues launching debug daemonset in OCP 4.12 related to security constraints

[ramperher]

Today, we have launched the first OCP 4.12 daily job and we have tested some workloads on top of it, including the workloads we use ot test CNF Cert Suite. You can see that job here.

Digging into the tnf results, we observed that there were a lot of unexpected results (some tests failed or were skipped and they should have passed), all of them caused by debug daemonset wrong behavior.

Taking a look into the events that happened in the cluster (it can be seen here, in events.txt file), we can see these issues related to the debug daemonset:

default                                            32m         Warning   FailedCreate                                      daemonset/debug                                                       Error creating: pods "debug-rtcj7" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
openshift-kube-controller-manager                  32m         Normal    CreatedSCCRanges                                  pod/kube-controller-manager-master-2                                  created SCC ranges for tnf namespace
default                                            32m         Warning   FailedCreate                                      daemonset/debug                                                       Error creating: pods "debug-8l6pg" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
default                                            32m         Warning   FailedCreate                                      daemonset/debug                                                       Error creating: pods "debug-dpkrr" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
default                                            32m         Warning   FailedCreate                                      daemonset/debug                                                       Error creating: pods "debug-2rqvh" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
default                                            32m         Warning   FailedCreate                                      daemonset/debug                                                       Error creating: pods "debug-ghjw9" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
default                                            32m         Warning   FailedCreate                                      daemonset/debug                                                       Error creating: pods "debug-rf2l8" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
default                                            32m         Warning   FailedCreate                                      daemonset/debug                                                       Error creating: pods "debug-cg58w" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
default                                            32m         Warning   FailedCreate                                      daemonset/debug                                                       Error creating: pods "debug-kk22c" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
default                                            32m         Warning   FailedCreate                                      daemonset/debug                                                       Error creating: pods "debug-nnhnd" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
default                                            5m2s        Warning   FailedCreate                                      daemonset/debug                                                       (combined from similar events): Error creating: pods "debug-r5rsl" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

My impression is that it is hitting some security issues caused by OCP 4.12. In theory, there are several changes in terms of security policies in OCP 4.12 (e.g. https://connect.redhat.com/en/blog/important-openshift-changes-pod-security-standards), and I suppose the debug daemonset specification should be reviewed to deal with them.

[sebrandon1]

@ramperher If I push up a branch, can you point to it and test?

[ramperher]

@ramperher If I push up a branch, can you point to it and test?

Sure, I can give a try in an OCP 4.12 cluster :) I'll take a look!

[ramperher]

@sebrandon1 , as discussed in #548, I think we don't need to change anything in the debug daemonset specification, after testing in OCP 4.12. It's just a matter of adding these two labels to default namespace (where debug daemonset is deployed):

              pod-security.kubernetes.io/enforce: privileged
              pod-security.kubernetes.io/enforce-version: latest

However, as discussed in the PR, I feel that, maybe, it is too extreme to set such privileges in default namespace, which can be used by other workloads, utilities, etc. Having said that, would it be a problem if debug daemonset is deployed in a specific namespace rather than in default namespace? Just to isolate the permissions we give to specific namespaces.

Then, second question I have: can it be included, in the README, this particularity about labelling the namespace where debug daemonset is deployed when working in OCP 4.12? Or should we wait until OCP 4.12 is finally released?

[sebrandon1]

I believe this change covers the security policies needed for the daemonset:
redhat-best-practices-for-k8s/privileged-daemonset#52

We can close this until we it any more (if any) problems.