Skip to content

feat(engine): node containment actuator — cordon + co-resident default-deny (ADR-0040) - #322

Merged
thejefflarson merged 2 commits into
mainfrom
thejefflarson/jef-731-node-containment-actuator-cordon-co-resident-default-deny
Aug 2, 2026
Merged

feat(engine): node containment actuator — cordon + co-resident default-deny (ADR-0040)#322
thejefflarson merged 2 commits into
mainfrom
thejefflarson/jef-731-node-containment-actuator-cordon-co-resident-default-deny

Conversation

@thejefflarson

Copy link
Copy Markdown
Owner

Summary

ADR-0040 step 1c (build-settled, docs/adr/0040-node-scoped-containment-mechanism-escalation.md §4–5): the ContainNode actuator half. The trigger + proposal surface (the boundary_break predicate, contain_node_link, the menu-resolver escalation) already landed on main in #321; this PR adds the render path, the deterministic rails, and their actuation metrics — all shadow-only, unit-tested, exercised nowhere live.

  • Cordon/uncordon renderers (respond::actuator::node_containment::render_cordon/render_uncordon): a Node.spec.unschedulable patch carrying a fixed ownership annotation (protector.jeffl.es/cordoned-by), applied via server-side apply under the protector field manager — a revert only ever lifts a cordon protector itself placed, never a human's or the autoscaler's. ContainNode joins the actuator render allowlist (previously render_isolation returned None for it).
  • Co-resident default-deny sweep (co_resident_denies): one QuarantineWorkload mitigation per labelled co-resident pod, reusing quarantine_workload_link's exact self-reference shape (refactored to take (node, labels) instead of a chain-derived QuarantineTarget, so both the chain-based quarantine and the node-containment sweep share one builder) and therefore render_isolation's existing renderer unchanged. An unlabelled co-resident pod declines exactly like any other quarantine candidate.
  • Deterministic rails (cordon_decision/revert_decision), pure over a NodeFact fleet, each unit-tested: never cordon a control-plane node; at most one node cordoned concurrently; refuse a cordon that would leave fewer than two schedulable workers; ownership-gated revert (a node lacking protector's annotation is never uncordoned).
  • Actuation metrics: a protector.engine.contain_node OTLP counter by event (proposed/applied/reverted/rail_refused) with a reason label on refusals (control-plane/one-node-cap/worker-floor/unlabelled/not-owned), mirrored to the existing in-cluster sink (no new egress). Wired to fire proposed from the real, already-live boundary_break trigger today; applied/reverted/rail_refused are ready but not yet reachable in production (see DECISION NEEDED below).
  • NodeContainmentActuator: the cluster-facing apply/revert glue a future break-glass/self-revert-verification ticket can call — cordons/uncordons the host and drives the co-resident sweep through the existing IsolationActuator path.

ContainNode remains impossible to auto-apply: no node arming rung is wired (that's ADR-0040 §6, a separate ticket), and is_additive_live() == false routes every ContainNode mitigation to Decision::Forbidden before the rails are even consulted — asserted by decide_forbids_contain_node_even_when_the_class_would_be_enabled.

Scope boundaries respected

  • No charts/ change, no RBAC write grant, no ../cluster touch (verified — git diff --stat -- charts/ is empty).
  • No PROTECTOR_*_ENABLE toggle added.
  • Shadow-default preserved: an audit install behaves byte-identically.

DECISION NEEDED: node read-RBAC + live wiring is a follow-up

The rails need node role/schedulability metadata (NodeFact: control-plane role, current schedulability, ownership-annotation state). Observing Kubernetes Node objects needs a new nodes get/list/watch RBAC grant this ticket deliberately does not add (explicitly out of scope). Rather than wire the rails against a fabricated "no data" fleet — which would silently default every rail to PASS, weakening exactly the protection ADR-0040's build-settled note warns against keeping at the floor of 2 — I left cordon_decision/revert_decision fully implemented and unit-tested, but not called from Engine::process's live per-pass loop. The proposed metric event is wired (it needs no node-fleet data — it mirrors the already-live boundary_break trigger). The natural next step: the ticket that adds nodes read RBAC to the chart is also the right place to wire a PlacementAdapter-style metadata-only Node observation and call these rails from the per-pass loop, lighting up applied/reverted/rail_refused for real. Flagging for the architect to ratify or redirect.

Testing

  • cargo fmt --all, cargo clippy --all-targets -- -D warnings — clean.
  • cargo test — 1179 lib tests pass (0 failed), including 17 new node_containment tests (renderers, co-resident sweep, every rail, metric-reason vocabulary), 3 new co_resident_workloads graph-walk tests, 1 new decide() invariant test, and 1 new metrics smoke test.
  • New tests would fail without this change (cover renderers, rails, the co-resident sweep, and the never-auto-applies invariant).
  • One pre-existing, unrelated flaky test in main.rs (enforce_rung_env_var_selects_the_ladder_position) — env-var mutation racing across parallel test threads; passes reliably with --test-threads=1; file untouched by this diff.
  • soundcheck:pr-review — no Critical/High findings (internal engine logic over typed K8s evidence, no injection/SSRF/CSRF/secrets surface).
  • /simplify — single self-review pass; factored a shared scheduled_on_host graph-walk helper out of self_severance/co_resident_workloads to remove duplication; otherwise clean.

Closes JEF-731

thejefflarson and others added 2 commits August 2, 2026 16:17
…t-deny (ADR-0040)

Implements ADR-0040 step 1c's actuator half (the trigger/proposal surface
already landed in #321): the ContainNode render path, its deterministic
rails, and their actuation metrics — all shadow-only, unit-tested.

- Cordon/uncordon renderers (`node_containment::render_cordon`/
  `render_uncordon`): a Node.spec.unschedulable patch carrying a fixed
  ownership annotation (protector.jeffl.es/cordoned-by), applied via
  server-side apply so a revert only ever lifts a cordon protector itself
  placed.
- Co-resident default-deny sweep (`co_resident_denies`): one
  QuarantineWorkload mitigation per labelled co-resident pod, reusing
  quarantine_workload_link's exact self-reference shape (refactored to take
  (node, labels) instead of a chain-derived QuarantineTarget) and therefore
  render_isolation's existing renderer — an unlabelled pod declines exactly
  like every other quarantine candidate.
- Deterministic rails (cordon_decision/revert_decision), pure over a
  NodeFact fleet: never cordon a control-plane node, at most one node
  cordoned concurrently, refuse a cordon leaving fewer than two schedulable
  workers, and ownership-gated revert.
- Actuation metrics: a `contain_node` counter by event
  (proposed/applied/reverted/rail_refused) with a reason label for refusals,
  mirrored to the existing in-cluster OTLP sink. Wired to fire "proposed"
  from the real, already-live boundary_break trigger; applied/reverted wait
  on the rung-3 wiring ticket.
- NodeContainmentActuator: the cluster-facing apply/revert glue a future
  break-glass/self-revert ticket can call, exercising the cordon + co-
  resident sweep together.

ContainNode remains impossible to auto-apply: no `node` arming rung is
wired, and is_additive_live() == false routes every mitigation to
Decision::Forbidden regardless of the rails — asserted by test.

Node role/schedulability observation (a `nodes` get/list/watch RBAC grant)
is a deliberate follow-up, not this ticket — see the node_containment
module doc for why evaluating the rails against fabricated "no data" would
be less safe than not wiring them into the live per-pass loop yet.

Closes JEF-731

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
…ip rail (ADR-0040)

revert() now takes the target's NodeFact and short-circuits to Actuation::DryRun
unless revert_decision(target).is_ok(), so the ownership gate (never lift a cordon
protector did not place) cannot be bypassed by a forgetful caller. Previously the
gate lived only in a doc-comment contract; the break-glass/self-revert path (ADR-0040
§6) will call this exact method, so the highest-blast action is now safe by
construction rather than by caller discipline. No live caller today; defense-in-depth
hardening of the path the next ticket builds on.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
@thejefflarson
thejefflarson merged commit 1162dbb into main Aug 2, 2026
5 checks passed
@thejefflarson
thejefflarson deleted the thejefflarson/jef-731-node-containment-actuator-cordon-co-resident-default-deny branch August 2, 2026 23:36
thejefflarson added a commit that referenced this pull request Aug 3, 2026
…ADR-0040 §5) (#323)

Break-glass and the standard ledger self-revert (ADR-0017/ADR-0036) previously only
knew the NetworkPolicy cut shape: routing a standing ContainNode reversion through the
generic network actuator would silently no-op (wrong object kind/name) and leave the
node cordoned. Adds a NodeContainmentRevert seam so Engine::process's self-revert loop
calls the real cordon-lift + co-resident-deny-lift path instead, gated on an attached
actuator and an observed NodeFact (skips rather than fabricates when either is missing,
matching node_containment's existing "no data" discipline). The ownership self-gate
(revert_decision, from #322) is exercised end-to-end.

Adds engine-level acceptance tests exercising the real proof/menu-resolution pipeline
(boundary_break trigger (c)) with the standing cut synthesized directly into the action
log, since neither the node arming rung nor node observation exist yet (a separate
ticket per ADR-0040 §6).


Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant