Add secret protection

[mkimuram]

What type of PR is this?

This PR adds secret-protection for protecting secret from deletion while it is in-use.

/kind feature

What this PR does / why we need it:

In the current k8s implementation, secret can be deleted while it is still used by other resources.
This will cause such issues that volume is failed to delete because secret needed for delete the volume is deleted before volume deletion.

Which issue(s) this PR fixes:

Fixes #101130

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Deletion of a secret is blocked until all pods and PVs that are using the secret are deleted.   

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

@mkimuram: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mkimuram]

This initial implementation works for below three cases:

1. Unused case: It should be deleted immediately.
2. Used by pod case: It should block deletion of secret until all pods using the secret are deleted.
3. Used by CSI PV case: It should block deletion of secret until all PVs using the secret are deleted.

We might need to discuss further on what other cases should be covered in issue/KEP, first. However, I'm sharing this working codes to make discussion easier.

[Unused case]
It should be deleted immediately.

kubectl create secret generic test-secret --from-literal='username=my-app' --from-literal='password=39528$vdg7Jb'
kubectl get secret test-secret -o yaml
kubectl delete secret test-secret

[Used by pod case]
It should block deletion of secret until all pods using the secret are deleted.

kubectl create secret generic test-secret --from-literal='username=my-app' --from-literal='password=39528$vdg7Jb'
kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/pods/inject/secret-pod.yaml
kubectl describe pod secret-test-pod
kubectl delete secret test-secret
kubectl get secret test-secret -o yaml
kubectl delete pod secret-test-pod
kubectl get secret

[Used by CSI PV case]
It should block deletion of secret until all PVs using the secret are deleted.
(Below assumes that csi-hostpath driver has already been installed.)

kubectl create secret generic test-secret --from-literal='username=my-app' --from-literal='password=39528$vdg7Jb'

cat << 'EOF' | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-hostpath-sc-with-secret
provisioner: hostpath.csi.k8s.io
parameters:
  csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
  csi.storage.k8s.io/node-publish-secret-name: test-secret
EOF

cat << 'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pv-claim
spec:
  storageClassName: csi-hostpath-sc-with-secret
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

kubectl get pv -o yaml | grep -A 5 " csi:"
    csi:
      driver: hostpath.csi.k8s.io
      nodePublishSecretRef:
        name: test-secret
        namespace: default
      volumeAttributes:

kubectl delete secret test-secret
kubectl get secret
kubectl delete pvc test-pv-claim
kubectl get secret

[mkimuram]

It is interesting to see [sig-storage] Projected secret optional updates should be reflected in volume [NodeConformance] [Conformance] test fails here. We will need to consider if secrets are actually used this way, and if so, we need to handle such use cases properly. (Allow force delete or provide ways to opt-in/opt-out this feature for particular secrets with labels/annotations.)

[fedebongio]

/remove-sig api-machinery

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mkimuram
To complete the pull request process, please assign deads2k, saad-ali after the PR has been reviewed.
You can assign the PR to them by writing /assign @deads2k @saad-ali in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• cmd/kube-controller-manager/OWNERS
• pkg/controller/volume/OWNERS
• pkg/volume/util/OWNERS
• plugin/pkg/admission/OWNERS
• plugin/pkg/auth/authorizer/OWNERS
• test/e2e/storage/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sftim]

If I have an immutable Secret and need to rotate this secret (eg because it has leaked), I think that it's important that I can force removal even if the Secret is in use.
Because of that likely need somewhere and the urgency of carrying it out, I recommend explicitly documenting the process to forcibly remove such a secret well before this change reaches beta (it's not enough to generically document how to forcibly bypass finalizers and hope that readers can solve the puzzle).

[k8s-ci-robot]

@mkimuram: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kubernetes-node-e2e	bb6854f	link	/test pull-kubernetes-node-e2e
pull-kubernetes-conformance-kind-ipv6-parallel	bb6854f	link	/test pull-kubernetes-conformance-kind-ipv6-parallel
pull-kubernetes-conformance-kind-ga-only-parallel	bb6854f	link	/test pull-kubernetes-conformance-kind-ga-only-parallel
pull-kubernetes-e2e-kind-ipv6	bb6854f	link	/test pull-kubernetes-e2e-kind-ipv6
pull-kubernetes-unit	bb6854f	link	/test pull-kubernetes-unit
pull-kubernetes-integration	bb6854f	link	/test pull-kubernetes-integration
pull-kubernetes-e2e-kind	bb6854f	link	/test pull-kubernetes-e2e-kind
pull-kubernetes-verify	bb6854f	link	/test pull-kubernetes-verify
pull-kubernetes-e2e-gce-ubuntu-containerd	bb6854f	link	/test pull-kubernetes-e2e-gce-ubuntu-containerd

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[jsafrane]

I think this deserves a KEP, with all use cases outlined and discussed on higher level than code.

[fedebongio]

/remove-sig api-machinery

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[mkimuram]

/remove-lifecycle rotten

This feature won't be implemented in-tree, but will be implemented as an external controller. However, keeping this open to mark that I'm still working on.

[k8s-ci-robot]

@mkimuram: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[dims]

Is this PR still needed, please rebase if so (or we can close it?)

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.