Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller to support k8s 1.25+

[pmalek]

Kubernetes 1.25 removes support for PodSecurityPolicy: https://kubernetes.io/blog/2022/08/23/kubernetes-v1-25-release/#pod-security-changes which we currently use in kong chart.

In order to support versions 1.25 and higher we should migrate to use PodSecurity Admission Controller as described in https://kubernetes.io/docs/tasks/configure-pod-container/migrate-from-psp/.

[randmonkey]

By reading kubernetes documents related to pod security admission, this could not be fully supported in helm. Using built in PodSecurity Admission Controller need to label the release namespace, but helm does not support it: helm/helm#3503 (which received a lot of -1s).
IMO we could just create a pod admission controller, and print a message to note user to label the namespace after installing.

[rainest]

Should we actually handle any new resources on our end for the new system? I coulda sworn I'd written down notes for this earlier, but cannot find them. Reviewing the feature again suggests we probably don't need to.

The docs suggest that there's a single admission controller per cluster: you make one when you're enabling the feature and it's used for enforcing policies on all namespaces after. We shouldn't need to create our own as such.

Since PodSecurity is enforced at the namespace level, not the ServiceAccount level, we probably shouldn't touch it unless we're sure that we're the sole namespace tenant, and the chart hasn't made that assumption before (even if it's probably commonly true).

Furthermore, PodSecurity doesn't afford the granular permission control offered by PodSecurityPolicy. You can only use the pre-set privileged/baseline/restricted levels. We can reasonably just test if we run into issues at baseline or restricted, determine if we can work to comply with a stricter set, and then document which levels we can run under. We shouldn't strictly need to add the enforcement information ourselves, as we already control the Pods and can modify them to require/not require permissions with or without the enforcement mechanism.

The chart managing its own security policy always seemed a bit dubious to me (I'd normally expect cluster admins to choose their allowed permissions and then require we comply with them, rather than us choosing the permissions that we already comply with), but someone asked for us to create it and you could at least make a reasonable argument that users aren't experts on which of the many permissions we needed--having the pre-set policy at least demonstrated a reasonable default for a complex piece of configuration. We don't need that so much with the simpler PodSecurity levels.