Fine-grained SupplementalGroups control

[everpeace]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add new API surface to control and track how supplemental groups are applied in the container.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3619-supplemental-groups-policy
• Discussion Link: Can bypass PodSecurityContext.SupplementalGroups by custom container image although PSP(or other policy engines) enforces the field kubernetes#112879
• Primary contact (assignee):　@everpeace
• Responsible SIGs: sig-node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.31
  • Beta release target (x.y): 1.32
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    • KEP-3169: Fine-grained SupplementalGroups control #3620
    • KEP-3619: update Test Plan and Graduation Criteria for KEP freeze #3862
    • KEP-3619: Cleanup After Freeze #3874
    • KEP-3619: update the latest milestone to v1.31 #4628
    • KEP-3619: Add SupplementalGroupsPolicy feature fields in Kubernetes API(Node.Status) and CRI(RuntimeStatusResponse) #4728
  • Code (k/k) update PR(s):
    • KEP-3619: Fine-grained SupplementalGroups control kubernetes#117842
    • KEP-3619: Add NodeStatus.Features.SupplementalGroupsPolicy API and e2e kubernetes#125470
  • Docs (k/website) update PR(s):
    • [dev-1.31] KEP-3619: Fine-grained SupplementalGroups control website#46920
    • blog post for KEP-3619: Fine-grained SupplementalGroups control website#46921
  • contained (optional):
    • KEP-3619: Fine-grained SupplementalGroups control containerd/containerd#9737
    • Implement RuntimeStatus.features.supplemental_groups_policy from KEP-3619 containerd/containerd#10410
  • CRI-O (optional):
    • server/*: add fine-grained SupplementalGroups control for enhanced security cri-o/cri-o#8268
    • KEP-3619: implement RuntimeStatus.features.supplemental_groups_policy cri-o/cri-o#8386
  • cri-tools (optional):
    • KEP-3619: Fine-grained SupplementalGroups control kubernetes-sigs/cri-tools#1438
• Beta
  • KEP (k/enhancements) update PR(s):
    • KEP-3619: updated KEP with alpha release status, and added haircommander as a reviewer #4809
    • KEP-3619: updated Production Readiness Review Questionnaire for beta release #4895
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/assign
/sig node
/kind feature

[k8s-ci-robot]

@everpeace: The label(s) /remove-label sig/scheduling cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, lead-opted-in, tracked/no, tracked/out-of-tree, tracked/yes. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/remove-label sig/scheduling

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[everpeace]

/remove-lifecycle stale

[SergeyKanzhelev]

/milestone v1.27

let's see if we can start it in 1.27

[SergeyKanzhelev]

/label lead-opted-in

[SergeyKanzhelev]

/stage alpha

[npolshakova]

Hello @everpeace 👋, 1.27 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for 1.27 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.27
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

It looks like #3620 will address most of these issues.

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[SergeyKanzhelev]

@npolshakova, kep yaml has correct values for stage, milestone, etc:

enhancements/keps/sig-node/3619-supplemental-groups-policy/kep.yaml

Lines 8 to 30 in 472a381

	status: implementable
	creation-date: 2022-10-14
	reviewers:
	- "@thockin"
	- "@mrunalp"
	- "@SergeyKanzhelev"
	approvers:
	- "@mrunalp"
	see-also: []
	replaces: []
	# The target maturity stage in the current dev cycle for this KEP.
	stage: alpha
	# The most recent milestone for which work toward delivery of this KEP has been
	# done. This can be the current (upcoming) milestone, if it is being actively
	# worked on.
	latest-milestone: "v1.27"
	# The milestone at which this feature was, or is targeted to be, at each stage.
	milestone:
	alpha: "v1.27"

This KEP updates to the latest template and covers testing section: #3862

PRR review completed:

enhancements/keps/prod-readiness/sig-node/3619.yaml

Lines 1 to 6 in 472a381

	# The KEP must have an approver from the
	# "prod-readiness-approvers" group
	# of http://git.k8s.io/enhancements/OWNERS_ALIASES
	kep-number: 3619
	alpha:
	approver: "@johnbelamaric"

So once #3862 is merged, this KEP is ready for the milestone

[npolshakova]

Great! Looks like #3862 went in so this enhancement as tracked for v1.27.
Thanks!

/label tracked/yes

[rashansmith]

Hi @everpeace,

👋 from the v1.31 Communications Team! We'd love for you to opt in to write a feature blog about your enhancement!
Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and is graduating.

To opt in, let us know and open a Feature Blog placeholder PR against the website repository by 3rd July, 2024. For more information about writing a blog see the blog contribution guidelines.

Note: In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[everpeace]

@LaurentGoderre

Docs team here. Can you create a PR for the documentation for this enhancement?

Thanks, I opened a draft PR for this KEP: kubernetes/website#46920

[everpeace]

@rashansmith

To opt in, let us know and open a Feature Blog placeholder PR against the website repository by 3rd July, 2024. For more information about writing a blog see the blog contribution guidelines.

Thanks, I'm happy to write a blog for this KEP. I opened a placeholder PR: kubernetes/website#46921

[ArkaSaha30]

Hey again @everpeace 👋, Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

Regarding this enhancement, it appears that there are currently 2 open PRs in the k/k repository out of which one is merged and the other is up for review.

For this KEP, we would need to do the following:

• Ensure the remaining PR is prepared for merging (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

If you anticipate missing code freeze, you can file an exception request in advance.

The status of this enhancement is currently marked as at risk for code freeze.

[everpeace]

Just checking in as we approach code freeze at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.
Regarding this enhancement, it appears that there are currently 2 open PRs in the k/k repository out of which one is kubernetes/kubernetes#117842 and the other is up for kubernetes/kubernetes#125470 (review) review).

@thockin @mrunalp @SergeyKanzhelev (as KEP reviwers/approvers)
cc/ @haircommander (as an active reviwer on this KEP PRs)

I apologize for bothering you again. As you may already be aware, my last k/k PR kubernetes/kubernetes#125470 (review) for this KEP is ready for review. I would be very grateful if this KEP could reach alpha in v1.31🙇.

[ArkaSaha30]

Hey again @everpeace 👋, 1.31 Enhancements team here,

Just a quick friendly reminder as we approach code freeze in about 2 days, at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

The current status of this enhancement is marked as at risk for code freeze. A few requirements mentioned in the comment #3619 (comment) still need to be completed. The following PR as per the description still needs to be merged:

• Implement RuntimeStatus.features.supplemental_groups_policy from KEP-3619 containerd/containerd#10410

If you anticipate missing code freeze, you can file an exception request in advance.

[thockin]

Whether the containerd PR is merged or not shouldn't matter here - their release train is async to ours anyway. @ArkaSaha30 tell me if you disagree?

[ArkaSaha30]

Whether the containerd PR is merged or not shouldn't matter here - their release train is async to ours anyway.

Yes, agree. Thank you for the clarification.
Since the code PRs to k/k are in place and merged, I am marking this enhancement as tracked for code freeze for the 1.31 Code Freeze!

[everpeace]

Thanks, I updated this Issue's description to make containerd/CRI-O PRs optional.

updated: all the optional PR are merged now 👍 The remaining part is just doc and blog.

[haircommander]

@everpeace do you plan on making progress on this in 1.32?

[kannon92]

Sounds like there is plans to bring this up for beta graduation in 1.32. Moving it to "Proposed for consideration".

[everpeace]

@haircommander

do you plan on making progress on this in 1.32?

Yes, I'm planning some improvements in 1.32 toward beta:

• error handling
  • rejecting pods with the SupplementalGroupsPolicy field on the node with Node.Status.Features.SupplementalGroupsPolicy: false (currently, silently falls back to Merge behavior)
• observability
  • add kube-state-metrics node_status_features{feature="SupplementalGroupsPolicy"}???

@kannon92

Moving it to "Proposed for consideration".

Thanks!

---

By the way, about the actual timing of its beta promotion, I think it would be better after both containerd and CRI-O supports SupplementalGroupsPolicy feature as defined in Beta graduation criteria.

• CRI-O already supports this feature in v1.31, but
• containerd has not supported this feature yet (we have to wait for containerd v2)

Thus, I'm not sure v1.32 is good timing for its beta promotion because I don't have information about containerd v2 release date even the expected one.

[tjons]

Hi, enhancements lead here - I inadvertently added this to the 1.32 tracking board 😀. Please readd it if you wish to progress this enhancement in 1.32.

/remove-label lead-opted-in

[everpeace]

@haircommander I'm now preparing PRR of this KEP for beta release in #4895. May this KEP lead-opt-in for v1.32?? (Sorry for the last minutes)

[haircommander]

/milestone v1.32
/stage beta

Thanks for the heads up @everpeace !

[haircommander]

/label lead-opted-in

[tjons]

Hello @everpeace 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting for stage beta for v1.32 (correct me, if otherwise).

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

To track this KEP for enhancements freeze:

• the KEP readme needs to have the rest of the PRR review questions filled out: see https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template for beta requirements.
• KEP-3619: updated Production Readiness Review Questionnaire for beta release #4895 will need to merge
• Please mark the status of the KEP as implementable for latest-milestone: v1.32.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!