PodSecurity admission (PodSecurityPolicy replacement)

[tallclair]

Enhancement Description

• One-line enhancement description (can be used as a release note): Introduce a new admission controller for enforcing the Pod Security Standards on pods in a namespace.

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/2579-psp-replacement/

• Discussion Link: https://docs.google.com/document/d/1woLGRoONE3EBVx-wTb4pvp4CI7tmLZ6lS26VTbosLKM/view#bookmark=id.km06bp3uzuco

• Primary contact (assignee): @tallclair

• Responsible SIGs: sig-auth, sig-security

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.22
  • Beta release target (x.y): 1.23
  • Stable release target (x.y): 1.25

• Alpha (1.22)

  • KEP (k/enhancements) update PR(s): #2582
  • Code (k/k) update PR(s):
    • kubernetes/kubernetes#103560
    • kubernetes/kubernetes#103558
    • kubernetes/kubernetes#103552
    • kubernetes/kubernetes#103543
    • kubernetes/kubernetes#103517
    • kubernetes/kubernetes#103465
    • kubernetes/kubernetes#103445
    • kubernetes/kubernetes#103382
    • kubernetes/kubernetes#103378
    • kubernetes/kubernetes#103365
    • kubernetes/kubernetes#103364
    • kubernetes/kubernetes#103341
    • kubernetes/kubernetes#103340
    • kubernetes/kubernetes#103326
    • kubernetes/kubernetes#103325
    • kubernetes/kubernetes#103323
    • kubernetes/kubernetes#103316
    • kubernetes/kubernetes#103315
    • kubernetes/kubernetes#103314
    • kubernetes/kubernetes#103099
    • kubernetes/kubernetes#103617
    • kubernetes/kubernetes#103452
  • Docs (k/website) update PR(s):
    • kubernetes/website#28903

• Beta (1.23)

  • KEP (k/enhancements) update PR(s):
    • #2895
    • #3035
  • Code (k/k) update PR(s):
    • kubernetes/kubernetes#106118
    • kubernetes/kubernetes#106097
    • kubernetes/kubernetes#106089
    • kubernetes/kubernetes#106087
    • kubernetes/kubernetes#106017
    • kubernetes/kubernetes#105959
    • kubernetes/kubernetes#105923
    • kubernetes/kubernetes#105922
    • kubernetes/kubernetes#105908
    • kubernetes/kubernetes#105898
    • kubernetes/kubernetes#105889
    • kubernetes/kubernetes#105857
    • kubernetes/kubernetes#105749
    • kubernetes/kubernetes#105485
    • kubernetes/kubernetes#105457
    • kubernetes/kubernetes#105314
    • kubernetes/kubernetes#105180
    • kubernetes/kubernetes#104588
    • kubernetes/kubernetes#104365
    • kubernetes/kubernetes#104217
    • kubernetes/k8s.io#3029
    • kubernetes/kubernetes#106106
  • Docs (k/website) update(s):
    • kubernetes/website#30225
    • kubernetes/website#30351
    • kubernetes/website#30422
    • kubernetes/website#30502

• Stable (1.25)

  • KEP (k/enhancements) update PR(s): #3310
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

[gracenng]

Hi @tallclair 👋 1.22 Enhancements shadow here.

This enhancement is in good shape for 1.22, a couple minor change requests in light of Enhancement Freeze on Thursday May 13th:

• KEP has not been merged to master
• the "reviewers" section in kep.yaml is not filled out
• <<[unresolved]>> in README
• Graduation Criteria in README

Thank you!

[tallclair]

• I'm nagging the approvers daily, and still expect to get this merged by enhancement freeze
• Filled in reviewers
• The only remaining unresolved sections are explicitly flagged as non-blocking for the alpha. Is it OK to leave these sections for beta, or will they cause tooling issues?
• Filled in graduation criteria

[gracenng]

Hi there, thanks for the speedy updates.

• I talked to the team and it is okay to have unresolved sections for alpha
• With regards to graduation criteria, the KEP template has been updated to clarify this. Apologies for the confusion.

[gracenng]

Hi @tallclair 👋 1.22 Enhancements shadow here.
I just wanted to double check to see if SIG-Auth will need to do anything for this enhancement and if so, are they OK with it?
Thanks!

[liggitt]

yes, this is one of the top three items sig-auth is tackling this release

[ritpanjw]

Hello @tallclair 👋 , 1.22 Docs Shadow here.

This enhancement is marked as Needs Docs for 1.22 release.
Please follow the steps detailed in the documentation to open a PR against dev-1.22 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Fri July 9, 11:59 PM PDT.
Also, take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thank you!

[gracenng]

Hi @tallclair 🌞 1.22 enhancements shadow here.

In light of Code Freeze on July 8th, this enhancement current status is at risk as there is no k/k PR tracked.
Please update the associated PR or let me know if there is none required for this enhancement.

Thanks

[liggitt]

kubernetes/kubernetes#103099 is open for this enhancement

[liggitt]

all code PRs are merged for beta, beta docs PR for 1.23 open at kubernetes/website#30351

[zetaab]

I am currently checking the PodSecurity alpha version and one question comes to my mind. I am using kube-prometheus (https://github.com/prometheus-operator/kube-prometheus) to monitor my cluster. When I execute kubectl label --dry-run=server --overwrite ns --all pod-security.kubernetes.io/enforce=restricted or kubectl label --dry-run=server --overwrite ns --all pod-security.kubernetes.io/enforce=baseline. I can see that node-exporter daemonset needs privileged mode. However, none of the other components needs that. With current design it is not possible to add different policy to single daemonset/deployment/sts? Only option is either 1) allow all pods to use privileged (imo not great option) in monitoring namespace 2) move node-exporter daemonset to another namespace that has privileged access. Which one we should choose in future?

[rensx5514]

I am currently checking the PodSecurity alpha version and one question comes to my mind. I am using kube-prometheus (https://github.com/prometheus-operator/kube-prometheus) to monitor my cluster. When I execute kubectl label --dry-run=server --overwrite ns --all pod-security.kubernetes.io/enforce=restricted or kubectl label --dry-run=server --overwrite ns --all pod-security.kubernetes.io/enforce=baseline. I can see that node-exporter daemonset needs privileged mode. However, none of the other components needs that. With current design it is not possible to add different policy to single daemonset/deployment/sts? Only option is either 1) allow all pods to use privileged (imo not great option) in monitoring namespace 2) move node-exporter daemonset to another namespace that has privileged access. Which one we should choose in future?

Obviously, I think it's option 2

[rensx5514]

I am currently checking the PodSecurity alpha version and one question comes to my mind. I am using kube-prometheus (https://github.com/prometheus-operator/kube-prometheus) to monitor my cluster. When I execute kubectl label --dry-run=server --overwrite ns --all pod-security.kubernetes.io/enforce=restricted or kubectl label --dry-run=server --overwrite ns --all pod-security.kubernetes.io/enforce=baseline. I can see that node-exporter daemonset needs privileged mode. However, none of the other components needs that. With current design it is not possible to add different policy to single daemonset/deployment/sts? Only option is either 1) allow all pods to use privileged (imo not great option) in monitoring namespace 2) move node-exporter daemonset to another namespace that has privileged access. Which one we should choose in future?

Why does node-exporter daemonset need privileged？It's just get metrics .

[zetaab]

@rensx5514 it collects metrics data from the node itself, manifest https://gist.github.com/zetaab/cae34a02d474ae98644ffa23b0815039

when trying to restricted policy to it:

Warning: node-exporter-v9zm9: host namespaces, hostPath volumes, hostPort, restricted volume types
Warning: node-exporter-64x4b: host namespaces, hostPath volumes, hostPort, restricted volume types
Warning: node-exporter-ffkw9: host namespaces, hostPath volumes, hostPort, restricted volume types
Warning: node-exporter-7ldf6: host namespaces, hostPath volumes, hostPort, restricted volume types
Warning: node-exporter-6fmbm: host namespaces, hostPath volumes, hostPort, restricted volume types
Warning: node-exporter-dxb5v: host namespaces, hostPath volumes, hostPort, restricted volume types

[rensx5514]

@rensx5514 it collects metrics data from the node itself, manifest https://gist.github.com/zetaab/cae34a02d474ae98644ffa23b0815039

when trying to restricted policy to it:

Warning: node-exporter-v9zm9: host namespaces, hostPath volumes, hostPort, restricted volume types
Warning: node-exporter-64x4b: host namespaces, hostPath volumes, hostPort, restricted volume types
Warning: node-exporter-ffkw9: host namespaces, hostPath volumes, hostPort, restricted volume types
Warning: node-exporter-7ldf6: host namespaces, hostPath volumes, hostPort, restricted volume types
Warning: node-exporter-6fmbm: host namespaces, hostPath volumes, hostPort, restricted volume types
Warning: node-exporter-dxb5v: host namespaces, hostPath volumes, hostPort, restricted volume types

I think maybe you can move node-exporter to kube-system. Otherwise, you might need to put it in a separate namespace.

[liggitt]

@zetaab @rensx5514 yes, the granularity of the PodSecurity feature is at the namespace level, so the namespace enforcement level must be the upper bound. Workload authors within the namespace can choose not to make use of all the allowed permissions, so you could certainly run a baseline- or restricted-compatible workload in a privileged namespace.

(as an aside, this issue is primarily meant as a tracking issue for the feature status... for questions or discussion about the feature, I'd recommend the sig-auth slack channel / mailing list / or bi-weekly meeting)

[liggitt]

plan is to continue beta work for 1.24 and target GA in 1.25

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[tallclair]

/remove-lifecycle stale

[marosset]

Can we try our best to make sure kubernetes/kubernetes#105919 lands in 1.25 too?

We are pursuing having 'PodOS' field go to GA in 1.25 too and without this change users may be unable to schedule Windows pods that specify a OS=windows with restricted policies.

For example:
Restricted policy enforces that ALL capabilities must be dropped for containers but pods that set OS=windows will be rejected if the specify anything in the capabilities fields.

@ravisantoshgudimetla FYI

[liggitt]

yes, I'd expect kubernetes/kubernetes#105919 to be required for GA of the PodOS feature

[ravisantoshgudimetla]

Can we try our best to make sure kubernetes/kubernetes#105919 lands in 1.25 too?

We are pursuing having 'PodOS' field go to GA in 1.25 too and without this change users may be unable to schedule Windows pods that specify a OS=windows with restricted policies.

For example: Restricted policy enforces that ALL capabilities must be dropped for containers but pods that set OS=windows will be rejected if the specify anything in the capabilities fields.

@ravisantoshgudimetla FYI

Yes, that is the goal and this was the PR I mentioned during sig-windows community meeting on Tuesday. We also explicitly mentioned it in KEP

Pod Security Standards are to be changed in 1.25 timeframe to accommodate the supported kubelet and kube-apiserver skew.