Skip to content

fix(engine): arm the node class — propose-only ContainNode, scope-confined, CP fail-closed - #324

Merged
thejefflarson merged 3 commits into
mainfrom
thejefflarson/jef-732-arm-the-node-class-armingrungnode-rung-3-posture-derived
Aug 3, 2026
Merged

fix(engine): arm the node class — propose-only ContainNode, scope-confined, CP fail-closed#324
thejefflarson merged 3 commits into
mainfrom
thejefflarson/jef-732-arm-the-node-class-armingrungnode-rung-3-posture-derived

Conversation

@thejefflarson

@thejefflarson thejefflarson commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Summary

Closes JEF-732.

Makes protector's ContainNode node-containment class eligible to propose, per
docs/adr/0040-node-scoped-containment-mechanism-escalation.md §5–6:

  • Arming ladder rung 3, node (arming_ladder.rs) — strictly above quarantine
    (edge-cut < quarantine < node), arms ContainNode in addition to its narrower
    predecessors. Neither lower rung ever arms it.
  • Startup validation (engine + chart, defense-in-depth) — refuses enforceRung: node
    unless mode: enforce with a non-empty enforceScope, mirroring the existing
    enforce-with-empty-scope refusal.
  • Node observation adapter (observe/adapter/node_fact.rs) — a metadata-only mapping
    from the watched Node fleet (name, control-plane signal, spec.unschedulable, the
    cordon-ownership annotation) to the rails' NodeFact shape. No .data read; status is
    never read either.
  • Posture-derived RBAC (charts/protector/templates/clusterrole.yaml) — nodes: [get, list, watch] always-on (metadata-only read, matching every other always-granted
    read); nodes: [patch] ONLY under mode: enforce + enforceRung: node, cluster-scoped.
  • Proposal-side rail wiring (Engine::process) — for every active ContainNode
    mitigation, evaluates the deterministic rails (node_containment::evaluate_proposal,
    wrapping cordon_decision) against the pass's observed NodeFact fleet and fires
    proposed/rail_refused metric events, failing closed on an unknown host.

REWORK (this PR was revised after review)

An initial version of this PR auto-cordoned once a ContainNode mitigation cleared the
rails + arming + scope. Review correctly found this violates ADR-0040 §5, which is
explicit: "a real node always has alive collateral, so the existing blast/alive-collateral
gate routes every node cut to human approval even at the armed rung; propose-first is
structural, not a toggle."
Fixed, along with two issues the same pass surfaced:

  1. CRITICAL — removed the auto-cordon path. evaluate_apply/ApplyOutcome (which
    called NodeContainmentActuator::apply) are replaced by evaluate_proposal/
    ProposalOutcome, which only ever decides whether to SURFACE a proposal. There is no
    code path in Engine that calls NodeContainmentActuator::apply at all now — only the
    REVERT half is wired (with_node_containment_actuator, from fix(engine): wire ContainNode into break-glass + ledger self-revert (ADR-0040 §5) #323, now pointed at the
    real cluster actuator). decide()'s generic Forbidden path for ContainNode
    (is_additive_live() == false) is untouched. New engine-level acceptance test
    (contain_node_never_reaches_an_apply_call_even_fully_eligible) proves this with a spy
    Actuator over a fully-eligible (armed, unscoped, rails-clean) scenario.
  2. HIGH — fixed an enforceScope bypass. ActuationScope::in_scope resolves a
    namespace per endpoint and returns vacuously true for ContainNode's host/<name>
    self-reference (which has none) — enforceScope confined nothing for this class.
    contain_node_in_scope confines it instead through the co-resident LABELLED workload
    set (co_resident_denies), which structurally includes the boundary-broken trigger
    (scheduled on the same host by construction). A host with no in-scope co-resident pod
    is never proposed.
  3. MEDIUM — control-plane detection fails closed. node_fact.rs now also recognizes
    the legacy node-role.kubernetes.io/master label and a control-plane-shaped taint
    (canonical or legacy key, any effect) as independent control-plane signals, unioned with
    the canonical label. (A node with none of the three signals stays the ordinary
    cordon-eligible worker default — the sound reading for vanilla/kubeadm clusters, where
    workers carry no role marker at all; documented inline as the judgment call.)
  4. MEDIUM — rewrote the chart/README/doc language describing enforceRung: node to
    state plainly that it makes ContainNode eligible to be proposed, never to
    auto-apply, at any rung.
  5. LOW — documented (my call) rather than implemented that the two-worker floor counts
    schedulable nodes, not Ready ones; cross-referencing pod-health-per-node was judged
    disproportionate scope for this fix, and Node.status stays unread per the observation
    adapter's metadata-only discipline.

Also rebased onto main (post-#323, which added the NodeContainmentRevert/
with_node_fact seam) and composed with it: Engine::process now refreshes the shared
node_facts map every pass from the real observation adapter (falling back to any
test-seeded fact when Snapshot::nodes is empty, so #323's existing tests are untouched),
and the real NodeContainmentActuator is wired as the revert actuator in run_loop.rs.

Design note, still holds: ContainNode gets its own PROPOSAL-side gate
(evaluate_proposal, called from Engine::process) rather than reusing the generic
decide() path, because decide() structurally forbids it forever
(is_additive_live() == false). Arming enforceRung: node only changes whether a
rails-clean cut is surfaced as an actionable proposal instead of a bare "not auto-enabled"
line — it never causes a cluster write.

Out of scope (a sibling ticket, JEF-733, merged as #323): the break-glass/self-revert
UNCORDON path.

Invariants preserved

  • Shadow-default byte-identical: mode: audit still renders zero nodes write grant
    (verified via helm template) and ContainNode is still Forbidden through the generic
    path regardless of rung, and now has no apply path at all.
  • No new PROTECTOR_*_ENABLE toggle — armed via mode + the existing ladder rung only.
  • 1000-line file cap respected (largest touched file is engine/src/engine/mod.rs at 969
    lines — worth watching, not yet over).
  • No ../cluster fork changes (that port is a separate ticket).
  • No ticket IDs in committed content — cites ADR-0040 throughout.

Test plan

  • cargo test (workspace): 1212 lib tests + 10 main.rs posture tests + 3 guard test
    binaries, all green. Covers: ArmingRung::Node arms ContainNode (and only that rung
    does); Posture::from_env refuses enforceRung=node without mode=enforce;
    observe_node_facts (metadata-only, legacy label, taint, unrelated-taint,
    no-signal-default); evaluate_proposal's full decision space (ineligible/none,
    fail-closed on unknown node, rail refusal surfaced, proposed with/without a standing
    cordon); contain_node_in_scope (unscoped, in-scope co-resident match, out-of-scope
    co-resident, unlabelled-only host); and the engine-level spy-actuator acceptance test
    proving no apply call is ever reachable.
  • Fixed a latent, pre-existing test race in main.rs's env-var-mutating posture tests
    (exposed by the third such test this PR adds) with a shared Mutex.
  • cargo fmt --check / cargo clippy --all-targets -- -D warnings: clean.
  • helm lint charts/protector: clean. helm template verified locally for all three
    postures (audit / enforce+quarantine / enforce+node) matching the CI assertions added to
    .github/workflows/chart.yml, including the enforceRung: node without mode: enforce
    refusal.
  • Ran /soundcheck:pr-review (manual, per-category) — no Critical/High findings. Ran a
    single-pass /simplify (Agent tool unavailable in this context) on the original diff,
    which folded a duplicated fleet lookup into the (now-removed) apply path.

🤖 Generated with Claude Code

thejefflarson and others added 2 commits August 2, 2026 23:42
…, live rail wiring (ADR-0040)

Rung 3 `node` (strictly above `quarantine`) arms ContainNode on the arming
ladder; the chart's ClusterRole gains an always-on metadata-only `nodes`
read (fleet observation for the rails) and a `nodes` patch write gated on
`mode: enforce` + `enforceRung: node`, mirroring the existing posture-derived
RBAC pattern. A new metadata-only Node watch (`observe::adapter::node_fact`)
feeds the deterministic rails a real NodeFact fleet each pass. Engine::process
now evaluates the rails (`node_containment::evaluate_apply`) against that
fleet for every active ContainNode mitigation and fires the `applied`/
`rail_refused` metric events, failing closed on a host absent from the fleet
(RailRefusal::UnknownNode) rather than fabricating a passing rail. Startup
(engine + chart) refuses `enforceRung: node` without `mode: enforce` + a
non-empty enforceScope, mirroring the existing empty-scope refusal.

The generic decide()/AutoApply path still forbids ContainNode unconditionally
(is_additive_live() == false); this ships ContainNode's own apply-side gate
instead, armed by the rung and safety-checked by the deterministic rails
rather than the generic blast-radius check. The break-glass/self-revert
uncordon path is out of scope here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
…-closed (ADR-0040 §5)

Rework of the node-class arming PR after review found ADR-0040 §5 violations:

1. CRITICAL — removed the auto-cordon path entirely. `evaluate_apply`/`ApplyOutcome`
   (which called `NodeContainmentActuator::apply` once armed+in-scope+rails-clean) are
   replaced by `evaluate_proposal`/`ProposalOutcome`, which only ever decides whether to
   SURFACE a proposal. ADR-0040 §5 is explicit that a node cut is propose-first by
   construction (alive collateral always present) — the deterministic rails are a bound
   on the human-approval gate, never a replacement for it. There is now no code path in
   Engine that calls `NodeContainmentActuator::apply` at all; only the REVERT half
   (`with_node_containment_actuator`, from #323) is wired. The `applied` metric event is
   gone; `proposed` now also fires (level-triggered) whenever the rails are rail-clean
   and armed, alongside the pre-existing edge-triggered ledger proposal event.

2. HIGH — fixed the enforceScope bypass. `ActuationScope::in_scope` resolves a namespace
   per endpoint and returns vacuously true for ContainNode's host self-reference (which
   has none), so `enforceScope` confined nothing for this class. `contain_node_in_scope`
   confines it instead through the co-resident labelled workload set (which structurally
   includes the boundary-broken trigger), reusing `co_resident_denies` + the existing
   per-mitigation `in_scope` check on each candidate — no scope bypass on the private
   internals, and a host with no in-scope co-resident pod is never proposed.

3. MEDIUM — control-plane detection now fails closed: recognizes the legacy
   `node-role.kubernetes.io/master` label and a control-plane-shaped taint (either
   canonical or legacy key, any effect) as additional, independent control-plane signals,
   union'd with the canonical label. A node with none of the three signals remains the
   ordinary cordon-eligible worker default (the correct reading for vanilla/kubeadm
   clusters, where workers carry no role marker at all).

4. MEDIUM — rewrote the chart/README language describing `enforceRung: node` and the
   engine's cut-severity ladder doc to state plainly that ContainNode is eligible to
   PROPOSE, never to auto-apply, at any rung.

5. LOW — documented (rather than implemented) that the two-worker floor counts
   schedulable nodes, not Ready ones; `Node.status` stays unread per the observation
   adapter's metadata-only discipline.

Rebased onto main (#323's NodeContainmentRevert/`with_node_fact` seam) and composed with
it: `Engine::process` now refreshes `node_facts` every pass from the real observation
adapter (falling back to test-seeded facts when `Snapshot::nodes` is empty), and the real
`NodeContainmentActuator` is wired as the revert actuator in `run_loop.rs`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
@thejefflarson
thejefflarson force-pushed the thejefflarson/jef-732-arm-the-node-class-armingrungnode-rung-3-posture-derived branch from f7a318b to d18c196 Compare August 3, 2026 06:58
@thejefflarson thejefflarson changed the title feat(engine): arm the node class — ArmingRung::Node, node observation, live rail wiring fix(engine): arm the node class — propose-only ContainNode, scope-confined, CP fail-closed Aug 3, 2026
…DR-0040)

The engine now runs an unconditional, metadata-only Node watch
(observe::adapter::node_fact::observe_node_facts, ADR-0040 §3/§6). The Helm
chart's ClusterRole was updated to grant nodes:[get,list,watch] always-on, but
the e2e harness deploys from its own inline ClusterRole in scripts/e2e.sh, which
was not. Without the grant the Node watch 403-floods the run loop every pass and
starves the deterministic proof passes, so the structural web->session-key chain
never proves within 300s and e2e fails.

Mirror the chart's always-on metadata-only read here. No `patch` grant: e2e never
arms enforceRung: node (ContainNode is propose-only at every rung, ADR-0040 §5),
so the posture-derived write surface stays absent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
@thejefflarson
thejefflarson merged commit 86bb0d3 into main Aug 3, 2026
6 checks passed
@thejefflarson
thejefflarson deleted the thejefflarson/jef-732-arm-the-node-class-armingrungnode-rung-3-posture-derived branch August 3, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant