Skip to content

feat: Allow Fileownership change through FSGroup and VOLUME_MOUNT_GROUP #1841

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

mytreya-rh
Copy link

What type of PR is this?
/kind feature

What this PR does / why we need it:
(As of now pls consider it as a draft PR to discuss the solution further.)
Allows the secrets to be mounted with FSGroup as specified in the POD spec.
Thus, A pod with a non-root user should be able to read a secret, and that secret need not be world-readable.

Which issue(s) this PR fixes :
Fixes #858

Is this a chart or deployment yaml update?
There is a yaml update for secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml (generated through make manifests).
It is added in the manifest_staging/deploy
But if this PR merges after: #1622, the change in SecretProviderClassPodStatusStatus won't be required anymore and we can revert the changes related to reconciler.

Special notes for your reviewer:

Problem:

Solution:

  • outline
  • Do the ownership change from within the driver by advertising the VOLUME_MOUNT_GROUP capability.

Notes:

  • The changes also include secret rotation based on SecretProviderClassPodStatusStatus, but will be reverted if feat: Use RequiresRepublish for secret rotation #1622 merges earlier
  • In addition, pulled up some of common repetitive code from the unit and e2e tests to make them a bit more terse

tests added in e2e-provider:

(leaving in the test status and runtime just for reference)
  • ok 16 Non-root POD with no FSGroup - create in 871ms
  • ok 17 Non-root POD with no FSGroup - Should fail to read non world readable secret in 186ms
  • ok 18 Non-root POD with no FSGroup - unmount succeeds in 10143ms
  • ok 19 Non-root POD with FSGroup - create in 1439ms
  • ok 20 Non-root POD with FSGroup - should read non world readable secret in 202ms
  • ok 21 Non-root POD with FSGroup - rotated secret should also be readable in 37119ms
  • ok 22 Non-root POD with FSGroup - unmount succeeds in 10177ms

unit tests:

  • nodeserver_test
    • TestNodePublishVolume_Errors/Invalid_FSGroup
    • TestNodePublishVolume/volume_mount_with_valid_FSGroup
  • reconciler_test
    • TestReconcileError/failed_to_parse_FSGroup
    • TestReconcileNoError/reconcile_with_FSGroup

TODOs:

  • squashed commits
  • includes documentation
  • adds unit tests

@k8s-ci-robot k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Jun 9, 2025
Copy link

linux-foundation-easycla bot commented Jun 9, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: mytreya-rh / name: Mytreya Kasturi (cbac857)

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jun 9, 2025
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If secrets-store-csi-driver contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jun 9, 2025
@k8s-ci-robot
Copy link
Contributor

Welcome @mytreya-rh!

It looks like this is your first PR to kubernetes-sigs/secrets-store-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/secrets-store-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 9, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @mytreya-rh. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot requested review from aramase and ritazh June 9, 2025 17:49
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mytreya-rh
Once this PR has been reviewed and has the lgtm label, please assign aramase for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jun 9, 2025
@enj enj moved this to Subprojects - Needs Triage in SIG Auth Jun 10, 2025
@enj enj added this to SIG Auth Jun 10, 2025
@dobsonj
Copy link
Member

dobsonj commented Jun 27, 2025

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 27, 2025
@mytreya-rh mytreya-rh force-pushed the allow_file_ownership branch from 898943d to b797f9d Compare June 29, 2025 16:08
@mytreya-rh
Copy link
Author

/retest

@enj enj moved this from Subprojects - Needs Triage to In Review in SIG Auth Jun 30, 2025
Copy link
Member

@aramase aramase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The windows job failures are related to this PR.

E0630 16:56:06.282028   10108 atomic_writer.go:419] "unable to change file with owner" err="chown c:\\var\\lib\\kubelet\\pods\\ff425598-c3fa-480d-a6af-814831673629\\volumes\\kubernetes.io~csi\\secrets-store-inline\\mount\\..2025_06_30_16_56_06.1168464543\\secretalias: not supported by windows" logContext="secrets-store-csi-driver" fullPath="c:\\var\\lib\\kubelet\\pods\\ff425598-c3fa-480d-a6af-814831673629\\volumes\\kubernetes.io~csi\\secrets-store-inline\\mount\\..2025_06_30_16_56_06.1168464543\\secretalias" owner=-1

ref: https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_secrets-store-csi-driver/1841/pull-secrets-store-csi-driver-e2e-windows/1939725228552753152/artifacts/2025-06-30T170106/secrets-store.log

@github-project-automation github-project-automation bot moved this from In Review to Changes Requested in SIG Auth Jun 30, 2025
@mytreya-rh mytreya-rh force-pushed the allow_file_ownership branch from b797f9d to cbac857 Compare June 30, 2025 19:43
@mytreya-rh
Copy link
Author

The windows job failures are related to this PR.

E0630 16:56:06.282028   10108 atomic_writer.go:419] "unable to change file with owner" err="chown c:\\var\\lib\\kubelet\\pods\\ff425598-c3fa-480d-a6af-814831673629\\volumes\\kubernetes.io~csi\\secrets-store-inline\\mount\\..2025_06_30_16_56_06.1168464543\\secretalias: not supported by windows" logContext="secrets-store-csi-driver" fullPath="c:\\var\\lib\\kubelet\\pods\\ff425598-c3fa-480d-a6af-814831673629\\volumes\\kubernetes.io~csi\\secrets-store-inline\\mount\\..2025_06_30_16_56_06.1168464543\\secretalias" owner=-1

ref: https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_secrets-store-csi-driver/1841/pull-secrets-store-csi-driver-e2e-windows/1939725228552753152/artifacts/2025-06-30T170106/secrets-store.log

Thanks @aramase !
Pushed a commit to skip Chown on Windows. Guess this is inline with FSGroup behavior as well on Windows nodes.

@mytreya-rh
Copy link
Author

/retest

Copy link
Member

@dobsonj dobsonj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one comment on test/bats/e2e-provider.bats, but otherwise LGTM. It's a useful fix, implementation looks correct, good test coverage, and passing CI tests. Netlify is warning about a line unrelated from your changes.

kubectl wait -n rotation --for=condition=Ready --timeout=60s pod ${curl_pod_name}
local pod_ip=$(kubectl get pod -n kube-system -l app=csi-secrets-store-e2e-provider -o jsonpath="{.items[0].status.podIP}")
run kubectl exec ${curl_pod_name} -n rotation -- curl http://${pod_ip}:8080/rotation?rotated=true
sleep 35 # 30 is poll interval, 5 second grace should be enough
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry that 35 seconds may not be enough to prevent flakes. In @test "Test auto rotation of mount contents and K8s secrets" (line 472) it used to sleep 60 seconds, but now it only sleeps 35 seconds? Is it possible for a reconcile loop to be delayed for some reason that would cause this to take longer than 35?

I would probably not reduce this below 60, we had one similar case in vault.bats waiting on secret rotation where we had to increase it to 120 to improve the pass rate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow file ownership to be set for secrets
4 participants