Support User Namespaces in pods

[derekwaynecarr]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add support for user namespaces in pods
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/127-user-namespaces/README.md
• Discussion Link: This was discussed in several sig-node meetings, most notable Nov 2 2021 and in this PR, that feedback was incorporated already by this KEP is taking: keps/127: Support User Namespaces #2101.
• Primary contact (assignee): @rata @giuseppe
• Responsible SIGs: sig-node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.25
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    • v1.24 127: Add KEP for user namespaces support #3065
    • v1.25 KEP-127: Mark as implementable (target phase I for 1.25) #3275
    • v1.27 KEP-127: Support userns in stateless pods with idmap mounts #3811
    • v1.28 [KEP-127] Add Pod Security Standards to User Namespaces KEP #4044
    • v1.28 KEP-127: add support for stateful pods #4084
    • v1.28 KEP-127: Fix user namespaces feature gate name #4107
    • v1.28 KEP-127: Update feature gate name and update last-milestone for 1.29 #4147
  • Code (k/k) update PR(s):
    • v1.27 KEP-127: user namespace support for stateless pods kubernetes#116377
    • v1.28 apis: drop check for volumes with user namespaces kubernetes#118691
    • v1.29 KEP-127: Update PSS based on feature gate kubernetes#118760
  • Docs (k/website) update PR(s):
    • v1.27 Document user namespace changes in v1.27 (KEP-127) website#39860
    • v1.28 Doc update for userns in 1.28 website#41908 (v1.28)
    • v1.29 [KEP-127] Add documentation about user namespaces and PSS website#43803 (v1.29)
• Beta
  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[derekwaynecarr]

This work is being done by @pweil- and is reviewed by @derekwaynecarr, it is sponsored by @kubernetes/sig-node

[mdshuai]

@derekwaynecarr Could you help create a user story card for this feature?

[idvoretskyi]

@derekwaynecarr can you confirm that this feature targets alpha for 1.5?

[pweil-]

@derekwaynecarr can you confirm that this feature targets alpha for 1.5?

Yes, this feature is experimental only so it would be considered alpha.

[idvoretskyi]

@derekwaynecarr @pweil- can you confirm that this item targets beta in 1.6?

[adelton]

@derekwaynecarr, the proposal kubernetes/kubernetes#34569 was closed by bot due to inactivity.

@pweil-, in kubernetes/kubernetes#34569 (comment) you've proposed the approach pweil-/kubernetes@16f29eb which changes the group of /var/lib/kubelet/pods to the remapped root group. Do I understand it correctly that this is currently not tracked in any pull request?

[adelton]

@pweil-, I also wonder if similar to docker's /var/lib/docker/<uid>.<gid> approach when --userns-remap is used, it might make sense to use /var/lib/kubelet/pods-<uid>.<gid> and just chown/chgroup everything in those subdirectories to the remapped <uid>.<gid>. Why did you opt for just the chgrp and not the full chown?

[pweil-]

@adelton in the end, I think having this be transparent to Kubernetes is the right approach. Whether that be something like shiftfs or implementation in the CRI (moby/moby#28593). You are correct that my existing proposal is not currently tracked in an open PR anymore.

The reasoning behind using the chgrp was to follow our fsgroup strategy where we just ensure group access instead of uid access.

[adelton]

Thanks @pweil-.

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

As for the fsgroup strategy, do you mean https://kubernetes.io/docs/concepts/policy/pod-security-policy/#fsgroup or some generic methodology within Kubernetes?

I have now filed kubernetes/kubernetes#55707 as an alternative approach where I make the remapped uid/gid an explicit option, and use those values to chown/chgrp the necessary directories.

[pweil-]

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

that would be ideal. Whether that is feasible (or more likely, feasible in an acceptable time frame) is another question 😄

As for the fsgroup strategy, do you mean https://kubernetes.io/docs/concepts/policy/pod-security-policy/#fsgroup or some generic methodology within Kubernetes?

Yes

I have now filed kubernetes/kubernetes#55707 as an alternative approach where I make the remapped uid/gid an explicit option, and use those values to chown/chgrp the necessary directories.

👍 subscribed

[adelton]

When you say transparent, you mean that nothing should be needed to be added to code or to configuration on Kubernetes' side to allow running under docker with userns-remap?

that would be ideal. Whether that is feasible (or more likely, feasible in an acceptable time frame) is another question

Ideally, the pod would specify how many distinct uids/gids it would require / list of uids it wants to see inside of the containers, and docker or different container runtime would setup the user namespace accordingly. But unless docker also changes ownership of the volumes mounted to the containers, Kubernetes will have to do that as part of the setup.

[adelton]

@pwel-, what is the best way to get some review and comments on kubernetes/kubernetes#55707, to get it closer to mergeable state?

[0xmichalis]

@pweil- ^

[pweil-]

@adelton I would try to engage the sig-node folks either at their Tuesday meeting or on slack: https://github.com/kubernetes/community/tree/master/sig-node

[adelton]

@derekwaynecarr, could you please bring kubernetes/kubernetes#55707 to sig-node's radar?

[idvoretskyi]

@pweil- @derekwaynecarr any progress on this feature is expected?

[npolshakova]

/remove-label lead-opted-in

[rata]

Can we add the 1.29 milestone for this PR?

We have for sure this PR targeted for 1.29: kubernetes/kubernetes#118760
We might do some KEP changes too.

[rata]

cc @saschagrunert @giuseppe @mrunalp @SergeyKanzhelev

[rata]

friendly ping @SergeyKanzhelev @mrunalp ?

[SergeyKanzhelev]

/milestone v1.29
/label lead-opted-in
/stage alpha

keeping the same stage as I understand

[salehsedghpour]

Hello @rata, @saschagrunert and @giuseppe 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on Friday, 6th October 2023.

This enhancement is targeting for stage alpha for 1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.29.
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

For this KEP, we would just need to update the following:

• Update the latest-milestone: 1.29 in the kep.yaml file

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[rata]

@salehsedghpour The PR with that is open here: #4147

[salehsedghpour]

@rata thank you, with this the status of this enhancement is marked tracked for enhancement freeze.

[drewhagen]

Hello @rata @giuseppe @saschagrunert 👋, v1.29 Docs Shadow here.
Does this enhancement work planned for v1.29 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023.
Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[saschagrunert]

I have the placeholder PR ready here: kubernetes/website#43437

[salehsedghpour]

Hey again @rata @giuseppe @saschagrunert 👋 Enhancements team here,

Just checking in as we approach code freeze at 01:00 UTC Wednesday 1st November.

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze:

• k/k/118760

With all this, the status of this KEP is at risk for code freeze.

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.
As always, we are here to help if any questions come up. Thanks!

[a-mccarthy]

hi @rata @giuseppe @saschagrunert, 👋 from the v1.29 Release Team-Communications! We would like to check if you have any plans to publish a blog for this KEP regarding new features, removals, and deprecations for this release.

If so, you need to open a PR placeholder in the website repository.
The deadline will be on Tuesday 14th November 2023 (after the Docs deadline PR ready for review)

Here is the 1.29 calendar

[rata]

@a-mccarthy I don't think so, but I'll talk with the others working on this and let you know ASAP if we change our minds :)

[vinayakankugoyal]

Hi all,
Great to see all the progress that has been made on this. I noticed that there is no released containerd version that supports idmap mounts. Which makes testing this feature hard for cloud-providers. containerd release-1.7 has a check that rejects any container that has UidMappings or GidMappings in a mount.

Would it be possible for containerd and other runtime support be made beta graduation blocking for this KEP?

/cc @mtaufen @liggitt

[salehsedghpour]

Hi @rata @giuseppe @saschagrunert,
As the following PR is now merged the status of this PR can now be marked with tracked for the code freeze.
kubernetes/kubernetes#118760

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP

[rata]

@salehsedghpour thanks! That should be all for this release! :)

@vinayakankugoyal CRIO and crun support this in stable releases already. For containerd the support is merged but coming in the 2.0 release, that it will take a while and I'm unsure how fast will cloud providers adopt it (do you have an idea for Google cloud? I'm afraid they will be very conservative and slow to adopt, and not sure it is worth waiting for them in that case). We were discussing on aiming for beta for 1.30, as there will be a containerd release by then probably, but we haven't decided yet. We will probably join the sig-node in some weeks to discuss it with the community.

[sftim]

https://github.com/kubernetes/enhancements/blob/fa647b424a0b11316e8e31274e50332b14434f9c/keps/sig-node/127-user-namespaces/README.md#pod-security-standards-pss-integration discusses integration with Pod Security Standards and puts a change behind a feature gate. That's hard to teach and I wonder if what we're really describing there is a feature gate for the admission checks.

The way I figure it, the standards themselves will only change exactly once: the point at which this new user namespaces feature graduates to stable.