set proper file permission for projected service account volume #89193

zshihang · 2020-03-17T23:08:33Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
implement the KEP kubernetes/enhancements#1598.

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[KEP]: https://github.com/kubernetes/enhancements/pull/1598

k8s-ci-robot · 2020-03-17T23:08:41Z

Hi @zshihang. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

zshihang · 2020-03-17T23:10:18Z

/assign @mikedanese @liggitt @tallclair

pkg/securitycontext/util.go

pkg/volume/util/operationexecutor/operation_generator.go

pkg/volume/volume.go

liggitt · 2020-03-19T14:21:52Z

pkg/volume/volume_linux.go

 	})
 }

-func changeFilePermission(filename string, fsGroup *int64, readonly bool, info os.FileInfo) error {


This is making significant changes in this method, and needs very close review by @kubernetes/sig-storage-pr-reviews

I don't have a good sense for the test coverage we have of all of the permissions permutations here, especially across different types of storage

/assign @jsafrane @msau42

/assign @gnufied
more eyes the better :-)

Is this behavior protected by any feature gate? When we added the new fsgroup change policy behavior, we guarded the change to this method with a feature gate so we could disable it if it caused regressions to existing behavior.

Regarding tests, we do have some testing fsgroup + projected volumes, but I don't think they cover the projected service account volume:

https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/projected_configmap.go#L59
https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/projected_downwardapi.go
https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/projected_secret.go

We don't have any e2es covering persistent volumes with runAsUser set (only fsgroup). And for fsgroup, it's only set in some basic tests like: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/testsuites/volumes.go#L177

I thought we had added fsgroup tests to subpath due to various regression issues, but I don't see fsgroup being set in those tests. I will dig further to figure out what happened to them.

mikedanese · 2020-03-20T14:40:26Z

cc @micahhausler

msau42 · 2020-03-20T16:09:41Z

/ok-to-test

pkg/api/v1/pod/util.go

zshihang · 2020-05-01T22:09:27Z

/retest

zshihang · 2020-05-01T22:26:05Z

/retest

msau42 · 2020-05-02T00:06:32Z

/lgtm

lavalamp · 2020-05-04T23:23:15Z

/approve

(Just propagating @mikedanese and @msau42 lgtm/approvals; I haven't actually reviewed this)

k8s-ci-robot · 2020-05-04T23:23:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lavalamp, msau42, zshihang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/OWNERS~~ [lavalamp]
~~test/e2e/auth/OWNERS~~ [lavalamp,msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2020-05-05T01:25:36Z

New changes are detected. LGTM label has been removed.

k8s-ci-robot · 2020-05-05T02:03:52Z

@zshihang: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-gce-iscsi	7cc5839331e75b70950e04dbbcade135b600fee7	link	`/test pull-kubernetes-e2e-gce-iscsi`
pull-kubernetes-e2e-aks-engine-azure	7cc5839331e75b70950e04dbbcade135b600fee7	link	`/test pull-kubernetes-e2e-aks-engine-azure`
pull-kubernetes-e2e-azure-file-windows	7cc5839331e75b70950e04dbbcade135b600fee7	link	`/test pull-kubernetes-e2e-azure-file-windows`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

zshihang · 2020-05-05T02:08:00Z

/retest

tallclair

Sorry I missed this earlier.

tallclair · 2020-10-14T19:03:23Z

pkg/volume/util/util.go

+	podutil.VisitContainers(&pod.Spec, podutil.InitContainers|podutil.Containers, func(container *v1.Container, containerType podutil.ContainerType) bool {
+		runAsUser, ok := securitycontext.DetermineEffectiveRunAsUser(pod, container)
+		// One container doesn't specify user or there are more than one
+		// non-root UIDs.


I think there might be a couple problems with this logic, if I'm reading it correctly:

A container could set the user in the container image. To get the image user, you can use the getImageUser(...) function, but you'd need to plumb through the runtime and export the method. Also note that the username can't be depended on.

A root user can only be ignored if it also has CAP_DAC_OVERRIDE (although it's included by default).

Even with these precautions, it's still possible that the container could drop permissions at runtime, but I think it should be acceptable to just cover that case by documentation (i.e. use FSGroup)

Actually, looks like I was misreading the code... I think this says that if the user isn't explicitly specified in all the containers (and the explicit users don't match) then return nil. In that case, I think the code is correct but the comment is incorrect (since it doesn't care about non-root).

using getImageUser would require a lot of changes in kubelet in terms of the image pulling and volume mounts.

Workaround kubernetes/kubernetes#82573 - this got fixed with kubernetes/kubernetes#89193 starting with Kubernetes 1.19

k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 17, 2020

k8s-ci-robot requested review from chakri-nelluri and cofyc March 17, 2020 23:11

zshihang force-pushed the master branch from 7799fe6 to d813472 Compare March 17, 2020 23:45

liggitt reviewed Mar 19, 2020

View reviewed changes

pkg/securitycontext/util.go Outdated Show resolved Hide resolved

liggitt reviewed Mar 19, 2020

View reviewed changes

pkg/volume/util/operationexecutor/operation_generator.go Outdated Show resolved Hide resolved

liggitt reviewed Mar 19, 2020

View reviewed changes

pkg/volume/volume.go Show resolved Hide resolved

liggitt reviewed Mar 19, 2020

View reviewed changes

k8s-ci-robot assigned jsafrane and msau42 Mar 19, 2020

zshihang force-pushed the master branch from d813472 to 2ae4ae5 Compare March 19, 2020 20:33

mikedanese added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Mar 20, 2020

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 20, 2020

mikedanese reviewed Mar 20, 2020

View reviewed changes

pkg/api/v1/pod/util.go Outdated Show resolved Hide resolved

k8s-ci-robot assigned gnufied Mar 20, 2020

mikedanese reviewed Mar 20, 2020

View reviewed changes

pkg/api/v1/pod/util.go Outdated Show resolved Hide resolved

mikedanese reviewed Mar 20, 2020

View reviewed changes

pkg/api/v1/pod/util.go Outdated Show resolved Hide resolved

mikedanese reviewed Mar 20, 2020

View reviewed changes

pkg/api/v1/pod/util.go Outdated Show resolved Hide resolved

k8s-ci-robot removed the kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API label May 1, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 2, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 4, 2020

set proper file permission for projected service account volume

3db7275

zshihang force-pushed the master branch from 00002f1 to 3db7275 Compare May 5, 2020 01:25

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 5, 2020

mikedanese added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 5, 2020

k8s-ci-robot merged commit 3365ed5 into kubernetes:master May 5, 2020

k8s-ci-robot added this to the v1.19 milestone May 5, 2020

hasheddan mentioned this pull request Jun 1, 2020

Docker images with non-root account fails to read token file aws/amazon-eks-pod-identity-webhook#8

Open

mikedanese mentioned this pull request Sep 16, 2020

Projected serviceAccountToken has mode of mounted file hardcoded to 0600 #82573

Closed

tallclair reviewed Oct 14, 2020

View reviewed changes

TheKangaroo mentioned this pull request Jun 16, 2021

controller won't run on k8s > 1.20 without fsGroup metallb/metallb#890

Closed

aramase mentioned this pull request Aug 26, 2021

Document issue with file mode in Kubernetes 1.18 Azure/azure-workload-identity#156

Closed

liggitt mentioned this pull request Nov 4, 2021

TOB-K8S-031: Adding credentials to containers by default is unsafe #81128

Closed

rfranzke added a commit to rfranzke/gardener that referenced this pull request Nov 30, 2021

Add workaround for projected service account tokens

bcdf7c2

Workaround kubernetes/kubernetes#82573 - this got fixed with kubernetes/kubernetes#89193 starting with Kubernetes 1.19

msau42 mentioned this pull request Nov 29, 2021

Ephemeral containers volume mount permissions differ from normal containers kubernetes/website#32351

Closed

krgostev pushed a commit to krgostev/gardener that referenced this pull request Apr 21, 2022

Add workaround for projected service account tokens

2eb4e07

Workaround kubernetes/kubernetes#82573 - this got fixed with kubernetes/kubernetes#89193 starting with Kubernetes 1.19

dimityrmirchev mentioned this pull request Jun 24, 2022

[GRM] Allow grm to read its service account token gardener/gardener#6187

Merged

krgostev pushed a commit to krgostev/gardener that referenced this pull request Jul 5, 2022

Add workaround for projected service account tokens

f17056a

Workaround kubernetes/kubernetes#82573 - this got fixed with kubernetes/kubernetes#89193 starting with Kubernetes 1.19

astraw99 mentioned this pull request Oct 23, 2022

Default v3 scaffold manager pod got permission denied error kubernetes-sigs/kubebuilder#3028

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

set proper file permission for projected service account volume #89193

set proper file permission for projected service account volume #89193

zshihang commented Mar 17, 2020 •

edited

Loading

k8s-ci-robot commented Mar 17, 2020

zshihang commented Mar 17, 2020 •

edited

Loading

liggitt Mar 19, 2020 •

edited

Loading

msau42 Mar 20, 2020

msau42 Mar 20, 2020

msau42 Mar 20, 2020

mikedanese commented Mar 20, 2020

msau42 commented Mar 20, 2020

zshihang commented May 1, 2020

zshihang commented May 1, 2020

msau42 commented May 2, 2020

lavalamp commented May 4, 2020

k8s-ci-robot commented May 4, 2020

k8s-ci-robot commented May 5, 2020

k8s-ci-robot commented May 5, 2020 •

edited

Loading

zshihang commented May 5, 2020

tallclair left a comment

tallclair Oct 14, 2020

tallclair Oct 14, 2020

zshihang Oct 14, 2020

set proper file permission for projected service account volume #89193

set proper file permission for projected service account volume #89193

Conversation

zshihang commented Mar 17, 2020 • edited Loading

k8s-ci-robot commented Mar 17, 2020

zshihang commented Mar 17, 2020 • edited Loading

liggitt Mar 19, 2020 • edited Loading

Choose a reason for hiding this comment

msau42 Mar 20, 2020

Choose a reason for hiding this comment

msau42 Mar 20, 2020

Choose a reason for hiding this comment

msau42 Mar 20, 2020

Choose a reason for hiding this comment

mikedanese commented Mar 20, 2020

msau42 commented Mar 20, 2020

zshihang commented May 1, 2020

zshihang commented May 1, 2020

msau42 commented May 2, 2020

lavalamp commented May 4, 2020

k8s-ci-robot commented May 4, 2020

k8s-ci-robot commented May 5, 2020

k8s-ci-robot commented May 5, 2020 • edited Loading

zshihang commented May 5, 2020

tallclair left a comment

Choose a reason for hiding this comment

tallclair Oct 14, 2020

Choose a reason for hiding this comment

tallclair Oct 14, 2020

Choose a reason for hiding this comment

zshihang Oct 14, 2020

Choose a reason for hiding this comment

zshihang commented Mar 17, 2020 •

edited

Loading

zshihang commented Mar 17, 2020 •

edited

Loading

liggitt Mar 19, 2020 •

edited

Loading

k8s-ci-robot commented May 5, 2020 •

edited

Loading