feat(chart): deny EtcdMember deletion at admission - #348
feat(chart): deny EtcdMember deletion at admission#348Andrei Kvapil (kvaps) wants to merge 2 commits into
Conversation
The docs have said since their first version that EtcdMember objects are created and deleted by the cluster controller and users should not touch them. Nothing enforced it, and the accident is unrecoverable: a member's PVC is controller-owned by it, so deleting the CR takes the data volume with it while the finalizer removes the member from etcd on the way out. Delete every member of a cluster and it dismembers itself with nothing left to restore from. One kubectl delete, a GitOps prune of an unexpected object, or a cleanup script sweeping CRs by label is enough. Install a ValidatingAdmissionPolicy denying DELETE on etcdmembers, except for the operator's ServiceAccount (scale-down and crash-loop replacement delete members deliberately), the garbage collector (an EtcdCluster deletion must still cascade), the namespace controller (namespace deletion must not hang), and anything in memberDeletionProtection.additionalAllowedUsers. Break-glass without uninstalling: annotate the member with etcd-operator.cozystack.io/allow-deletion=true. Requires Kubernetes 1.30+; memberDeletionProtection.enabled=false installs without the guard. Deliberately not gated on .Capabilities — that is empty for a plain helm template, so gating there would silently drop the policy from the rendered release manifests. Docs: state in the API model that deleting a member destroys its data (the sentence previously said only "create or edit"), document the guard and the break-glass path, add the annotation step to the broken-member recovery recipe it would otherwise block, and remove the policy before the CRDs during a manual teardown — deleting a CRD makes the apiserver delete every member, which the policy would deny. Assisted-By: Claude Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
github-actions
Bot
added
documentation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds a configurable Kubernetes ChangesEtcdMember deletion protection
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant KubernetesAPIServer
participant ValidatingAdmissionPolicy
participant EtcdMember
Client->>KubernetesAPIServer: DELETE EtcdMember
KubernetesAPIServer->>ValidatingAdmissionPolicy: Evaluate deletion request
ValidatingAdmissionPolicy->>EtcdMember: Read allow-deletion annotation
ValidatingAdmissionPolicy-->>KubernetesAPIServer: Allow or Deny
KubernetesAPIServer-->>Client: Deletion response
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@charts/etcd-operator/templates/member-deletion-policy.yaml`:
- Around line 71-73: Update the EtcdMember deletion policy binding around
policyName and validationActions so each release is scoped to its managed
namespace using matchResources/namespaceSelector, or explicitly prevent multiple
releases from installing the chart. Ensure legitimate EtcdMember deletions
remain allowed when multiple releases exist and avoid cluster-wide binding
collisions.
- Around line 32-37: Add "system:kube-controller-manager" to the allowed users
assembled by the member-deletion policy template alongside the existing
controller identities, and update the member-deletion policy tests to verify
this fallback identity is permitted.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 5d39716f-91cc-4694-bd72-f78cb90175ca
📒 Files selected for processing (6)
charts/etcd-operator/templates/member-deletion-policy.yamlcharts/etcd-operator/tests/member_deletion_policy_test.yamlcharts/etcd-operator/values.yamldocs/concepts.mddocs/installation.mddocs/operations.md
| {{- $allowed := concat | ||
| (list | ||
| (printf "system:serviceaccount:%s:%s" .Release.Namespace (include "etcd-operator.serviceAccountName" .)) | ||
| "system:serviceaccount:kube-system:generic-garbage-collector" | ||
| "system:serviceaccount:kube-system:namespace-controller") | ||
| .Values.memberDeletionProtection.additionalAllowedUsers }} |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg '(^|/)member-deletion-policy\.yaml$|etcd-operator|memberDeletionProtection' || true
echo
echo "member-deletion-policy.yaml candidates:"
fd -a 'member-deletion-policy\.yaml$' . || true
echo
echo "Search relevant symbols/config/vals:"
rg -n "memberDeletionProtection|member-deletion-policy|memberDeletion|additionalAllowedUsers|controller-manager|service-account|Service Account" -S . | head -n 200Repository: cozystack/etcd-operator
Length of output: 4803
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "member-deletion-policy.yaml:"
cat -n charts/etcd-operator/templates/member-deletion-policy.yaml
echo
echo "values memberDeletionProtection section:"
sed -n '26,40p' charts/etcd-operator/values.yaml | cat -n
echo
echo "docs concepts section around allowed users:"
sed -n '1,70p' docs/concepts.md | cat -n
echo
echo "chart tests section around member-deletion-policy assertions:"
sed -n '1,150p' charts/etcd-operator/tests/member_deletion_policy_test.yaml | cat -nRepository: cozystack/etcd-operator
Length of output: 17388
🌐 Web query:
Kubernetes kube-controller-manager use-service-account-credentials service account username system:kube-controller-manager authorizationrbac
💡 Result:
The --use-service-account-credentials flag for kube-controller-manager controls how individual control loops (controllers) authenticate with the Kubernetes API server [1][2]. When --use-service-account-credentials is set to true (default is false), the kube-controller-manager creates a dedicated service account for each built-in controller [3][2]. These controllers then use these individual service account credentials to perform their tasks [3][2]. The roles required for each controller are predefined in Kubernetes and are prefixed with system:controller: (e.g., system:controller:deployment-controller) [2]. In this mode, the kube-controller-manager's primary credential only requires the system:kube-controller-manager ClusterRole [2][4]. If --use-service-account-credentials is set to false (monolithic mode), the kube-controller-manager runs all its internal control loops using its own single credential [2][4]. In this configuration, that credential must be granted the system:kube-controller-manager ClusterRole as well as all the permissions required by every individual controller, which is not handled automatically by the cluster [2][4]. The system:kube-controller-manager user identity is a built-in system identity (User: system:kube-controller-manager) [5]. By default, this user is bound to the system:kube-controller-manager ClusterRole via a ClusterRoleBinding [5]. This role provides the base access required for the component to function, such as accessing essential resources like namespaces, endpoints, and secrets [2][6]. If the controller manager requires additional permissions (e.g., accessing ConfigMaps), administrators may need to update this ClusterRole or bind additional roles to the system:kube-controller-manager user [6].
Citations:
- 1: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
- 2: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
- 3: https://docs.datadoghq.com/security/default_rules/jbp-64r-kyz/
- 4: Document administration of credentials for system roles/bindings/groups kubernetes/kubernetes#44418
- 5: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-role-bindings.yaml
- 6: ClusterRole system:kube-controller-manager cannot get configmaps in v1.12.0-rc's kubernetes/kubernetes#68986
Allow the kube-controller-manager fallback identity.
Clusters running kube-controller-manager without --use-service-account-credentials authenticate deletion requests as system:kube-controller-manager, so cascade deletes from a deleted EtcdCluster and namespace cleanup can be rejected and leave objects stuck. Add system:kube-controller-manager to the allowed delete users and cover it in the member-deletion policy tests.
Proposed fix
(printf "system:serviceaccount:%s:%s" .Release.Namespace (include "etcd-operator.serviceAccountName" .))
"system:serviceaccount:kube-system:generic-garbage-collector"
"system:serviceaccount:kube-system:namespace-controller"
+ "system:kube-controller-manager")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| {{- $allowed := concat | |
| (list | |
| (printf "system:serviceaccount:%s:%s" .Release.Namespace (include "etcd-operator.serviceAccountName" .)) | |
| "system:serviceaccount:kube-system:generic-garbage-collector" | |
| "system:serviceaccount:kube-system:namespace-controller") | |
| .Values.memberDeletionProtection.additionalAllowedUsers }} | |
| {{- $allowed := concat | |
| (list | |
| (printf "system:serviceaccount:%s:%s" .Release.Namespace (include "etcd-operator.serviceAccountName" .)) | |
| "system:serviceaccount:kube-system:generic-garbage-collector" | |
| "system:serviceaccount:kube-system:namespace-controller" | |
| "system:kube-controller-manager") | |
| .Values.memberDeletionProtection.additionalAllowedUsers }} |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@charts/etcd-operator/templates/member-deletion-policy.yaml` around lines 32 -
37, Add "system:kube-controller-manager" to the allowed users assembled by the
member-deletion policy template alongside the existing controller identities,
and update the member-deletion policy tests to verify this fallback identity is
permitted.
| spec: | ||
| policyName: {{ include "etcd-operator.fullname" . }}-protect-members | ||
| validationActions: ["Deny"] |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg '(^|/)member-deletion-policy.yaml$|charts/|templates/'
echo
echo "Target file outline/stat:"
wc -l charts/etcd-operator/templates/member-deletion-policy.yaml
ast-grep outline charts/etcd-operator/templates/member-deletion-policy.yaml || true
echo
echo "Target file relevant contents:"
cat -n charts/etcd-operator/templates/member-deletion-policy.yaml | sed -n '1,140p'
echo
echo "Search for related templates/values/CRDs:"
find charts/etcd-operator -maxdepth 4 -type f -print | sort | xargs -r rg -n "allowlist|validationActions|policyName|etcd-operator\.fullname|validation|binding|Deny|EtcdMember|ServiceAccount|namespace|admission|restrict|member-deletion|protection" || trueRepository: cozystack/etcd-operator
Length of output: 47357
🌐 Web query:
Kubernetes ValidatingAdmissionPolicyBinding spec matchResources namespaceSelector no namespaceSelector applies namespace-scoped resources allNamespaces
💡 Result:
In Kubernetes, the namespaceSelector field within ValidatingAdmissionPolicyBinding (part of the matchResources configuration) utilizes a LabelSelector that defaults to an empty selector when unset [1][2][3]. An empty LabelSelector matches everything [1][4][3]. Therefore, if namespaceSelector is omitted in the ValidatingAdmissionPolicyBinding spec, it effectively does not filter by namespace, meaning the policy will apply to resources in all namespaces, provided they are otherwise matched by the policy's configuration [5][1][2]. Key points regarding this behavior: - Scope: The namespaceSelector decides whether to run the admission control policy based on whether the object's namespace matches the selector [1][2][3]. - Default Behavior: Because it defaults to an empty LabelSelector, omitting the field means no namespace-based filtering is applied, resulting in evaluation for all namespaces [1][4][3]. - Cluster-scoped Resources: For cluster-scoped resources, the policy is never skipped due to namespaceSelector [1][4][2]. - Interaction with Policy: The binding's matchResources is intersected with the ValidatingAdmissionPolicy's own matchConstraints [5]. Only requests matched by the underlying policy can be selected by the binding [5]. Do not confuse this with NetworkPolicy or other Kubernetes objects where the presence or absence of namespaceSelector may have different implicit behaviors (such as defaulting to the local namespace) [6]. In the context of ValidatingAdmissionPolicyBinding, an omitted namespaceSelector behaves as "match all" [1][3].
Citations:
- 1: https://kubernetes.io/docs/reference/kubernetes-api/definitions/match-resources-v1-admissionregistration/
- 2: https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/extension_apis/validatingadmissionpolicybinding-admissionregistration-k8s-io-v1
- 3: https://kubernetes.website.cncfstack.com/docs/reference/kubernetes-api/policy-resources/validating-admission-policy-binding-v1/
- 4: https://www.pulumi.com/registry/packages/kubernetes/api-docs/admissionregistration/v1/validatingadmissionpolicybinding/
- 5: https://kubernetes.io/docs/reference/kubernetes-api/admissionregistration/validating-admission-policy-binding-v1/
- 6: https://kodekloud.com/community/t/with-and-without-namespaceselector/339151
Scope the EtcdMember deletion policy binding per release namespace or enforce singleton installation.
The binding has no matchResources/namespaceSelector, so one release’s policy is applied cluster-wide to every EtcdMember. A second release installs another same-name policy whose allowed list excludes the first release’s ServiceAccount, which can block legitimate scale-down/failure-replacement deletes from that operator. Share one cluster-wide allowlist across releases, scope bindings to each managed namespace, or explicitly enforce singleton installation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@charts/etcd-operator/templates/member-deletion-policy.yaml` around lines 71 -
73, Update the EtcdMember deletion policy binding around policyName and
validationActions so each release is scoped to its managed namespace using
matchResources/namespaceSelector, or explicitly prevent multiple releases from
installing the chart. Ensure legitimate EtcdMember deletions remain allowed when
multiple releases exist and avoid cluster-wide binding collisions.
The Kamaji datastore test deletes EtcdMembers on purpose to exercise MemberRemove plus the GenerateName replacement. With member deletion denied at admission that request is rejected — correctly: the guard exists to stop exactly this command when it is issued by accident. Annotate the member with etcd-operator.cozystack.io/allow-deletion=true first, which is the documented path for a deliberate deletion. Done in the test rather than by exempting its identity in the policy, so the e2e stays honest about what a human has to do. Assisted-By: Claude Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Timofei Larkin (lllamnyp)
left a comment
Timofei Larkin (lllamnyp)
left a comment
There was a problem hiding this comment.
Changes requested — the direction is exactly right; four things need fixing before this ships, one of them empirical.
This is the guard I asked for on #347, and the hard parts are done well: Deny + failurePolicy: Fail, the break-glass annotation instead of identity exemptions for tests, the e2e replaying the human procedure rather than sidestepping it, the BYO-SA/fullname unittest, and concepts.md finally saying delete with the consequence spelled out. What follows is about the exemption list, the teardown story, and two pieces of framing.
1. The exemption list is incomplete for non-SA-credential control planes
CodeRabbit's first inline comment is correct and I want to second it explicitly: on clusters where kube-controller-manager runs without --use-service-account-credentials, the GC and namespace-cleanup deletes authenticate as the user system:kube-controller-manager, not as the per-controller ServiceAccounts. On such a cluster this policy wedges both EtcdCluster cascade deletion and namespace deletion — the two flows the allowlist exists to keep working. Add system:kube-controller-manager to the allowlist (harmless where per-controller SAs are in use) and cover it in the unittest alongside the other three.
2. Two releases of this chart deny each other's operators
CodeRabbit's second comment has the wrong details (policy names are fullname-prefixed, so there's no name collision) but the right defect: the binding carries no matchResources/namespaceSelector, so every release's policy applies cluster-wide to all etcdmembers, while each allowlist contains only its own release's SA. Admission is deny-wins across matching policies — install the chart twice and each operator's legitimate scale-down and crash-loop replacement is blocked by the other release's policy. That's not hypothetical: --watch-namespace (#341) exists precisely so multiple scoped operators can coexist. Either scope the binding (spec.matchResources.namespaceSelector) when the release is namespace-scoped, or document a hard singleton constraint — but the current combination of cluster-wide match and per-release allowlist can't stay.
3. The CRD-teardown story needs an empirical check — the mechanism as written is probably wrong
The template comment, values.yaml, and installation.md all state that deleting the CRD makes the apiserver delete every EtcdMember, the policy denies those requests, and the CRD hangs in Terminating. I don't believe the middle step. CRD-instance cleanup (customresourcecleanup.apiextensions.k8s.io) is performed by the apiextensions-apiserver in-process, through the storage layer rather than the authenticated request path — the same long-standing gap that keeps admission webhooks from firing for CRs deleted during CRD deletion. If that holds for VAP too (and I expect it does), then:
- the documented stuck-CRD scenario doesn't occur via the policy — if a CRD wedges during manual teardown, the cause is member finalizers with no operator left to process them, which is a pre-existing, separately documented mode; and
- more importantly, CRD deletion/replacement remains a member-destruction path the guard cannot see — and that scenario was on the original threat list this guard responds to.
Please verify once on kind: policy active, delete the CRD, observe whether the deletes are denied and what identity the audit log records. Then either keep the teardown step with a corrected rationale, or replace it with an honest known-limitation note ("the guard does not cover CRD-deletion-driven member removal"). Both outcomes are fine; an asserted mechanism nobody has observed is not.
4. Rewrite the no-.Capabilities rationale — the stated one doesn't hold
The comment justifies always-rendering with "Capabilities is empty for a plain helm template, so gating would silently drop the policy from the rendered release manifests." But helm template has --api-versions, and build-dist-manifests is our own CI — one flag in the Makefile keeps a gated policy in dist. As stated, the rationale is refutable, and someone will refute it and then gate the template.
The durable rationale is different, and worth stating instead: capability detection makes a security guard's absence silent. A live helm install against a pre-1.30 apiserver would quietly omit the policy; Argo-style GitOps renders, which pass the destination cluster's API versions, would do the same. For a control whose whole purpose is preventing an unrecoverable accident, absence must be loud and explicitly chosen — which is exactly what always-render plus memberDeletionProtection.enabled=false delivers. Right decision; please make the comment defend it with the argument that survives.
5. Drop "a GitOps prune" from the threat framing
It appears in the template comment and again in concepts.md. There is no legitimate scenario in which a GitOps tool manages EtcdMember objects — the only sanctioned non-operator writer is cmd/etcd-migrate, which creates and never deletes. Listing "a GitOps prune" alongside plausible accidents reads as if these objects might reasonably live in a desired-state repo. State the contract positively instead: nothing manages EtcdMembers declaratively, and a GitOps tool attempting this delete is a tracking misconfiguration the policy correctly stops — not a workflow to accommodate.
Minor
installation.md's manual-teardown commands hardcodeetcd-operator-protect-members, but the objects are named{{ fullname }}-protect-members. Correct for the default release name, wrong for any other — either note that, or show a label-based delete.- The e2e annotates via
Get+Update, which can 409 against a concurrent member-controller write and fail the test spuriously; aMergeFrompatch of the annotation is conflict-proof.
Bottom line
Fix the two allowlist/scoping defects (1, 2), run the kind experiment and make the teardown docs say what actually happens (3), and reframe the two comments (4, 5). With those, this closes the hole #347 argued about at the right layer, and I'm happy to approve. On #343: my position stands that it's worth keeping in draft as defense-in-depth for exactly the populations this chart flag can exempt (enabled=false, pre-1.30) — but that decision can wait until this lands.
2 participants
Implements the direction proposed in the review on #347: stop the event at the API boundary instead of trying to survive it in the controllers.
Why
docs/concepts.mdhas said since its first version thatEtcdMemberobjects are created and deleted by the cluster controller and users should not touch them. Nothing enforced it, and the accident it invites is unrecoverable:Delete-reclaim StorageClass;One
kubectl delete, a GitOps prune of an object the tool did not expect to be managed, or a cleanup script sweeping CRs by label is enough. This is not hypothetical — it is what happened on a dev cluster: theEtcdMemberCRs of seven namespaces were removed, the volumes went with them, and the running pods kept serving from already-deleted volumes until a node reboot ended them twelve days later.What
A
ValidatingAdmissionPolicyin the chart denyingDELETEonetcdmembers, withfailurePolicy: FailandvalidationActions: ["Deny"]. Allowed through:system:serviceaccount:kube-system:generic-garbage-collectorEtcdClustermust still cascade to its memberssystem:serviceaccount:kube-system:namespace-controllermemberDeletionProtection.additionalAllowedUsersBreak-glass without uninstalling the policy — annotate, then delete:
The denial message names the member, says the data is not recoverable from there, and points at scaling the
EtcdClusterinstead.Notes on two decisions
Not gated on
.Capabilities.ValidatingAdmissionPolicyis GA since 1.30, and my first draft failed the render when the API was absent. But.Capabilities.APIVersionsis empty for a plainhelm template, so that check silently dropped the policy frombuild-dist-manifestsoutput — exactly the manifest where its absence is least visible. Instead the policy always renders when enabled; on an older apiserver the install fails loudly withno matches for kind ValidatingAdmissionPolicy, andmemberDeletionProtection.enabled=falseis the documented way out.CRD teardown ordering. Removing the CRD makes the apiserver delete every
EtcdMember, and the policy denies those requests — leaving the CRD stuck inTerminating.helm uninstallremoves the policy first, so the normal path is fine; the teardown docs now spell it out for manual teardowns that skip Helm.Docs
concepts.md: the API-model sentence now says delete and states the consequence — this is the one-line fix asked for twice in review. Plus a new section documenting the guard, the allowed requesters, break-glass, the 1.30 requirement and the uninstall ordering.operations.md: the broken-member recovery recipe iskubectl delete etcdmember— which this policy blocks. Added the annotation step, and a note that typing it out is what separates a deliberate data-discarding recovery from the same command issued by accident.installation.md: teardown removes the policy before the CRDs.Tests
charts/etcd-operator/tests/member_deletion_policy_test.yaml— 8 cases: installed by default, binding names its policy, matches onlyDELETEonetcdmembers, denies rather than warns and fails closed, all three system requesters still allowed, break-glass annotation honoured, extra allowed users accepted, andenabled=falserenders nothing (for pre-1.30 clusters).helm unittest8/8;go test ./...unaffected and green.Relationship to the other two PRs
AllMembersLostterminal condition) — with member deletion denied, that state stops being reachable through supported paths. Happy to close it; if it survives as defence-in-depth, it needs the migrate-window fix noted in review (worth recording:cmd/etcd-migrateverifies both operator Deployments are scaled to zero unless--skip-controller-checkis passed, so the window is misuse-only rather than routine).Summary by CodeRabbit
New Features
EtcdMemberdeletion.Documentation
EtcdMemberedits/deletes, explained recovery using the “break-glass” annotation, and clarified Kubernetes version requirements.Tests