[Explore]: Supporting Pod Security in Kubernetes 1.25

[gcapizzi]

Background

Kubernetes 1.25 will remove PodSecurityPolicy, which we are still using. The Pod Security admission controller is its designated replacement, and we should migrate to it.

The new Pod Security admission controller works via Pod Security Standards that can be applied at the namespace level. The restricted standard seems to fit the Eirini PodSecurityPolicy, but we should take a deeper look at what adopting it would imply.

Acceptance Criteria

One or more stories/chores to migrate Korifi from PodSecurityPolicy to the Pod Security admission controller. Keep in mind that we need to do this for both LRPs/RunWorkloads and Tasks.

Timebox

No timebox

Dev Notes

• Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller
  • This provides a list of things to check before doing the switch - important!
• Pod Security Standards
• Pod Security Admission

[kieron-dev]

We tried annotating a cf space namespace with the labels:

pod-security.kubernetes.io/audit=restricted
pod-security.kubernetes.io/enforce=restricted

This enables pod security at the highest restricted level on the namespace.

Then we attempted to push an app. The kpack logs show that the security policy requires certain policies to be set on the pod spec, namely:

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault # or Localhost

We hacked a kpack build-pod spec and proved that adding these to the container and all init containers allowed the pod to be created.

[kieron-dev]

We are going to modify kpack to set these securityContext values.

You can build kpack images using ./hack/release.sh europe-west1-docker.pkg.dev/cf-on-k8s-wg/kpack foo.yml. Just remember to gcloud auth configure-docker europe-west1-docker.pkg.dev first and create the kpack repository in GCP artifacts.

Diff:

diff --git a/pkg/apis/build/v1alpha2/build_pod.go b/pkg/apis/build/v1alpha2/build_pod.go
index c6cd887..5270d71 100644
--- a/pkg/apis/build/v1alpha2/build_pod.go
+++ b/pkg/apis/build/v1alpha2/build_pod.go
@@ -526,7 +526,6 @@ func (b *Build) BuildPod(images BuildPodImages, buildContext BuildContext) (*cor
                                    return nil
                                }
                                return []string{fmt.Sprintf("-report=%s", ReportTOMLPath)}
-
                            }(),
                            b.Spec.Tags),
                        VolumeMounts: volumeMounts([]corev1.VolumeMount{
@@ -670,7 +669,6 @@ func (b *Build) notarySecretVolume() corev1.Volume {
                EmptyDir: &corev1.EmptyDirVolumeSource{},
            },
        }
-
    }

    return corev1.Volume{
@@ -860,7 +858,7 @@ func (b *Build) setupSecretVolumesAndArgs(secrets []corev1.Secret, filter func(s
            annotatedUrl := secret.Annotations[GITSecretAnnotationPrefix]
            args = append(args, fmt.Sprintf("-ssh-%s=%s=%s", "git", secret.Name, annotatedUrl))
        default:
-           //ignoring secret
+           // ignoring secret
            continue
        }

@@ -1108,6 +1106,18 @@ func steps(f func(step func(corev1.Container, ...stepModifier))) []corev1.Contai
        for _, m := range modifiers {
            container = m(container)
        }
+       fp := false
+       tp := true
+       container.SecurityContext = &corev1.SecurityContext{
+           AllowPrivilegeEscalation: &fp,
+           RunAsNonRoot:             &tp,
+           Capabilities: &corev1.Capabilities{
+               Drop: []corev1.Capability{"ALL"},
+           },
+           SeccompProfile: &corev1.SeccompProfile{
+               Type: corev1.SeccompProfileType("RuntimeDefault"),
+           },
+       }
        containers = append(containers, container)
    })
    return containers

[kieron-dev]

We applied the above change by copying the release yaml (foo.yml) to korifi/dependencies/kpack-release-0.5.2.yaml and deploying korifi on kind. We also made the artifact registry public. This will be deleted after this explore.

Now the kpack build succeeds, but the statefulset fails to create its pods:

Events:
  Type     Reason        Age                   From                    Message
  ----     ------        ----                  ----                    -------
  Warning  FailedCreate  86s (x16 over 4m10s)  statefulset-controller  create Pod cf-proc-4b3daf83-3e6f-479f-a91e-7f6d57eb-97d6eb2e8e-0 in StatefulSet cf-proc-4b3daf83-3e6f-479f-a91e-7f6d57eb-97d6eb2e8e failed error: pods "cf-proc-4b3daf83-3e6f-479f-a91e-7f6d57eb-97d6eb2e8e-0" is forbidden: violates PodSecurity "
restricted:latest": unrestricted capabilities (container "opi" must set securityContext.capabilities.drop=["ALL"])

So we need to change the eirini-controller pod spec to include those securityContext settings.

[kieron-dev]

We changed korifi to set the appropriate labels on cf-space namespaces:

diff --git a/controllers/controllers/workloads/shared.go b/controllers/controllers/workloads/shared.go
index 95db433..5110f2e 100644
--- a/controllers/controllers/workloads/shared.go
+++ b/controllers/controllers/workloads/shared.go
@@ -56,6 +56,8 @@ func createOrPatchNamespace(ctx context.Context, client client.Client, log logr.
                for key, value := range labels {
                        namespace.Labels[key] = value
                }
+               namespace.Labels["pod-security.kubernetes.io/enforce"] = "restricted"
+               namespace.Labels["pod-security.kubernetes.io/audit"] = "restricted"
                return nil
        })
        if err != nil {

and eirini to remove the PSPs and set the appropriate container security context:

diff --git a/k8s/stset/lrp_to_statefulset.go b/k8s/stset/lrp_to_statefulset.go
index b8015a3eb..6c961695e 100644
--- a/k8s/stset/lrp_to_statefulset.go
+++ b/k8s/stset/lrp_to_statefulset.go
@@ -107,6 +107,9 @@ func (c *LRPToStatefulSet) Convert(statefulSetName string, lrp *eiriniv1.LRP, pr
                        Ports:           ports,
                        SecurityContext: &corev1.SecurityContext{
                                AllowPrivilegeEscalation: &allowPrivilegeEscalation,
+                               Capabilities: &corev1.Capabilities{
+                                       Drop: []corev1.Capability{"ALL"},
+                               },
                        },
                        Resources:      getContainerResources(lrp.Spec.CPUWeight, lrp.Spec.MemoryMB, lrp.Spec.DiskMB),
                        LivenessProbe:  livenessProbe,

And we can successfully push dora and curl its endpoint.

[kieron-dev]

To complete, we need to:

• {Talk to / write a story to talk to} kpack team about setting container security contexts on their build-pods so they support the restricted context. Or perhaps allow the level to be configurable
• Write a story to accept a pod security level in korifi config, and apply that on cf-space namespaces
• Write a story to accept a pod security level in eirini-controller, and apply the appropriate container security context to make workload pods work in that context
• Write a story to apply the restricted pod security level to korifi controlled namespaces (-api, -controllers, ...).

[danail-branekov]

{Talk to / write a story to talk to} kpack team about setting container security contexts on their build-pods so they support the restricted context. Or perhaps allow the level to be configurable

buildpacks-community/kpack#972

Write a story to accept a pod security level in korifi config, and apply that on cf-space namespaces
Write a story to accept a pod security level in eirini-controller, and apply the appropriate container security context to make workload pods work in that context

After discussing the topic with @georgethebeatle we decided that we could initially only support the restricted standard. If we wanted to make this configurable, then we would need to change workloads' labels as part of the spaces reconciliation loop which might have unpredictable impact on running pods. Here is the story:

#1220

Write a story to apply the restricted pod security level to korifi controlled namespaces (-api, -controllers, ...).

#1221

The stories above have been grouped via the pod-security-standards label