feat(engine): node containment actuator — cordon + co-resident default-deny (ADR-0040) - #322
Merged
thejefflarson merged 2 commits intoAug 2, 2026
Conversation
…t-deny (ADR-0040) Implements ADR-0040 step 1c's actuator half (the trigger/proposal surface already landed in #321): the ContainNode render path, its deterministic rails, and their actuation metrics — all shadow-only, unit-tested. - Cordon/uncordon renderers (`node_containment::render_cordon`/ `render_uncordon`): a Node.spec.unschedulable patch carrying a fixed ownership annotation (protector.jeffl.es/cordoned-by), applied via server-side apply so a revert only ever lifts a cordon protector itself placed. - Co-resident default-deny sweep (`co_resident_denies`): one QuarantineWorkload mitigation per labelled co-resident pod, reusing quarantine_workload_link's exact self-reference shape (refactored to take (node, labels) instead of a chain-derived QuarantineTarget) and therefore render_isolation's existing renderer — an unlabelled pod declines exactly like every other quarantine candidate. - Deterministic rails (cordon_decision/revert_decision), pure over a NodeFact fleet: never cordon a control-plane node, at most one node cordoned concurrently, refuse a cordon leaving fewer than two schedulable workers, and ownership-gated revert. - Actuation metrics: a `contain_node` counter by event (proposed/applied/reverted/rail_refused) with a reason label for refusals, mirrored to the existing in-cluster OTLP sink. Wired to fire "proposed" from the real, already-live boundary_break trigger; applied/reverted wait on the rung-3 wiring ticket. - NodeContainmentActuator: the cluster-facing apply/revert glue a future break-glass/self-revert ticket can call, exercising the cordon + co- resident sweep together. ContainNode remains impossible to auto-apply: no `node` arming rung is wired, and is_additive_live() == false routes every mitigation to Decision::Forbidden regardless of the rails — asserted by test. Node role/schedulability observation (a `nodes` get/list/watch RBAC grant) is a deliberate follow-up, not this ticket — see the node_containment module doc for why evaluating the rails against fabricated "no data" would be less safe than not wiring them into the live per-pass loop yet. Closes JEF-731 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
…ip rail (ADR-0040) revert() now takes the target's NodeFact and short-circuits to Actuation::DryRun unless revert_decision(target).is_ok(), so the ownership gate (never lift a cordon protector did not place) cannot be bypassed by a forgetful caller. Previously the gate lived only in a doc-comment contract; the break-glass/self-revert path (ADR-0040 §6) will call this exact method, so the highest-blast action is now safe by construction rather than by caller discipline. No live caller today; defense-in-depth hardening of the path the next ticket builds on. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
thejefflarson
deleted the
thejefflarson/jef-731-node-containment-actuator-cordon-co-resident-default-deny
branch
August 2, 2026 23:36
7 tasks
thejefflarson
added a commit
that referenced
this pull request
Aug 3, 2026
…ADR-0040 §5) (#323) Break-glass and the standard ledger self-revert (ADR-0017/ADR-0036) previously only knew the NetworkPolicy cut shape: routing a standing ContainNode reversion through the generic network actuator would silently no-op (wrong object kind/name) and leave the node cordoned. Adds a NodeContainmentRevert seam so Engine::process's self-revert loop calls the real cordon-lift + co-resident-deny-lift path instead, gated on an attached actuator and an observed NodeFact (skips rather than fabricates when either is missing, matching node_containment's existing "no data" discipline). The ownership self-gate (revert_decision, from #322) is exercised end-to-end. Adds engine-level acceptance tests exercising the real proof/menu-resolution pipeline (boundary_break trigger (c)) with the standing cut synthesized directly into the action log, since neither the node arming rung nor node observation exist yet (a separate ticket per ADR-0040 §6). Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ADR-0040 step 1c (build-settled,
docs/adr/0040-node-scoped-containment-mechanism-escalation.md§4–5): theContainNodeactuator half. The trigger + proposal surface (theboundary_breakpredicate,contain_node_link, the menu-resolver escalation) already landed onmainin #321; this PR adds the render path, the deterministic rails, and their actuation metrics — all shadow-only, unit-tested, exercised nowhere live.respond::actuator::node_containment::render_cordon/render_uncordon): aNode.spec.unschedulablepatch carrying a fixed ownership annotation (protector.jeffl.es/cordoned-by), applied via server-side apply under theprotectorfield manager — a revert only ever lifts a cordon protector itself placed, never a human's or the autoscaler's.ContainNodejoins the actuator render allowlist (previouslyrender_isolationreturnedNonefor it).co_resident_denies): oneQuarantineWorkloadmitigation per labelled co-resident pod, reusingquarantine_workload_link's exact self-reference shape (refactored to take(node, labels)instead of a chain-derivedQuarantineTarget, so both the chain-based quarantine and the node-containment sweep share one builder) and thereforerender_isolation's existing renderer unchanged. An unlabelled co-resident pod declines exactly like any other quarantine candidate.cordon_decision/revert_decision), pure over aNodeFactfleet, each unit-tested: never cordon a control-plane node; at most one node cordoned concurrently; refuse a cordon that would leave fewer than two schedulable workers; ownership-gated revert (a node lacking protector's annotation is never uncordoned).protector.engine.contain_nodeOTLP counter byevent(proposed/applied/reverted/rail_refused) with areasonlabel on refusals (control-plane/one-node-cap/worker-floor/unlabelled/not-owned), mirrored to the existing in-cluster sink (no new egress). Wired to fireproposedfrom the real, already-liveboundary_breaktrigger today;applied/reverted/rail_refusedare ready but not yet reachable in production (see DECISION NEEDED below).NodeContainmentActuator: the cluster-facing apply/revert glue a future break-glass/self-revert-verification ticket can call — cordons/uncordons the host and drives the co-resident sweep through the existingIsolationActuatorpath.ContainNoderemains impossible to auto-apply: nonodearming rung is wired (that's ADR-0040 §6, a separate ticket), andis_additive_live() == falseroutes everyContainNodemitigation toDecision::Forbiddenbefore the rails are even consulted — asserted bydecide_forbids_contain_node_even_when_the_class_would_be_enabled.Scope boundaries respected
charts/change, no RBAC write grant, no../clustertouch (verified —git diff --stat -- charts/is empty).PROTECTOR_*_ENABLEtoggle added.auditinstall behaves byte-identically.DECISION NEEDED: node read-RBAC + live wiring is a follow-up
The rails need node role/schedulability metadata (
NodeFact: control-plane role, current schedulability, ownership-annotation state). Observing KubernetesNodeobjects needs a newnodesget/list/watchRBAC grant this ticket deliberately does not add (explicitly out of scope). Rather than wire the rails against a fabricated "no data" fleet — which would silently default every rail to PASS, weakening exactly the protection ADR-0040's build-settled note warns against keeping at the floor of 2 — I leftcordon_decision/revert_decisionfully implemented and unit-tested, but not called fromEngine::process's live per-pass loop. Theproposedmetric event is wired (it needs no node-fleet data — it mirrors the already-liveboundary_breaktrigger). The natural next step: the ticket that addsnodesread RBAC to the chart is also the right place to wire aPlacementAdapter-style metadata-only Node observation and call these rails from the per-pass loop, lighting upapplied/reverted/rail_refusedfor real. Flagging for the architect to ratify or redirect.Testing
cargo fmt --all,cargo clippy --all-targets -- -D warnings— clean.cargo test— 1179 lib tests pass (0 failed), including 17 newnode_containmenttests (renderers, co-resident sweep, every rail, metric-reason vocabulary), 3 newco_resident_workloadsgraph-walk tests, 1 newdecide()invariant test, and 1 new metrics smoke test.main.rs(enforce_rung_env_var_selects_the_ladder_position) — env-var mutation racing across parallel test threads; passes reliably with--test-threads=1; file untouched by this diff.soundcheck:pr-review— no Critical/High findings (internal engine logic over typed K8s evidence, no injection/SSRF/CSRF/secrets surface)./simplify— single self-review pass; factored a sharedscheduled_on_hostgraph-walk helper out ofself_severance/co_resident_workloadsto remove duplication; otherwise clean.Closes JEF-731