Skip to content

test(bench): add route-forwarded-backend fixture to cut-choice bench (ADR-0034) - #314

Merged
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-698-deployed-pod-bench-route-forwarded-backend-fixture
Aug 1, 2026
Merged

test(bench): add route-forwarded-backend fixture to cut-choice bench (ADR-0034)#314
thejefflarson merged 1 commit into
mainfrom
thejefflarson/jef-698-deployed-pod-bench-route-forwarded-backend-fixture

Conversation

@thejefflarson

Copy link
Copy Markdown
Owner

Summary

Adds one CASE to scripts/judge_bakeoff_cutchoice.py (the T2b cut-choice bench, ADR-0034): route_forwarded_entry_cve — a backend reached only via an Ingress route (no direct Service exposure) rendered in the ENTRY position with its own loaded-at-runtime CVE.

  • Node identity: workload/public/Pod/checkout-api-7f9d4c8b6d-x2p9k — distinct from the existing web-frontend-5d8 / web-cache-0 pair so it's unambiguously the route-forwarded case, not a rename of entry_only_log4j.
  • Fixture: mirrors entry_only_log4j's shape exactly (CVE loaded-at-runtime on the entry, clean downstream cache, same containment menu format) but represents a workload that only becomes an internet-facing entry because a route forwards to it.
  • Expected assessment: attack
  • Expected contain: {checkout-api-7f9d4c8b6d-x2p9k} only (the route-forwarded backend itself, mirroring the entry-only minimal-cut discipline)

This validates the now-merged Ingress exposure observer (feat(engine): ingress exposure observer — route-transitive Exposure::Internet through declared L7 routes (ADR-0038), #313): a route-forwarded backend becomes an Exposure::Internet ENTRY and its loaded CVE should promote through the same edge-CVE lane the judge already handles for a directly-exposed entry.

Test plan

  • python3 -c "..." dry-imports the script and confirms CASES grows from 9 → 10, and render() builds the prompt for the new case without error (no network/model call — see command below).
  • wc -l scripts/judge_bakeoff_cutchoice.py → 275 lines, well under the repo's 1,000-line hard cap.
  • cargo fmt --check → clean (no Rust files touched by this change).
  • /soundcheck:pr-review self-review → no Critical/High findings (pure data-literal test fixture addition, no new I/O/network/deserialization surface).
  • /simplify self-review (single-pass, no Agent-tool fan-out available in this context) → already clean; new case reuses existing menu()/CACHE/CACHE_LINE/OBJS helpers, no new code path.
python3 -c "
import importlib.util
spec = importlib.util.spec_from_file_location('bench', 'scripts/judge_bakeoff_cutchoice.py')
m = importlib.util.module_from_spec(spec)
spec.loader.exec_module(m)
assert len(m.CASES) == 10
case = next(c for c in m.CASES if c[0] == 'route_forwarded_entry_cve')
rendered = m.render(case)
assert m.ROUTE_ENTRY in rendered and 'CVE-2024-45519' in rendered
"

Pending (not run here, per ADR-0033 deployed-pod discipline): the AUTHORITATIVE deployed-judge validation — running this fixture against the cluster-deployed qwen3:1.7b via a port-forward (OLLAMA_URL=http://localhost:11436/api/chat python3 scripts/judge_bakeoff_cutchoice.py qwen3:1.7b), confirming it scores attack/correct cut-set, and confirming downstream_cve_only and downstream_behavioral show no drift. This requires cluster access this environment doesn't have and is expected to run in the main loop.

Closes JEF-698

…(ADR-0034)

Adds a route_forwarded_entry_cve CASE to judge_bakeoff_cutchoice.py: a
backend reached only via an Ingress route (no direct Service exposure),
rendered in the ENTRY position with its own loaded-at-runtime CVE.
Expected assessment=attack, contain=only that backend — mirroring
entry_only_log4j but with a distinct node identity
(checkout-api-7f9d4c8b6d-x2p9k) so it's clearly the route-forwarded case.

This validates that the Ingress exposure observer's Exposure::Internet
promotion of a route-forwarded backend feeds the T2b cut-choice prompt
through the same edge-CVE lane as a directly-exposed entry.

Confirmed locally: the script imports and CASES/render() build the
prompt without error (dry parse only, no model call). The AUTHORITATIVE
deployed-judge validation (does this fixture pass on qwen3:1.7b via the
cluster port-forward, and do downstream_cve_only / downstream_behavioral
still hold with no drift) is pending and runs separately against the
deployed judge.
@thejefflarson

Copy link
Copy Markdown
Owner Author

Deployed A/B validation (main loop, deployed qwen3:1.7b on pod pfwwl) — PASS:

[OK] route_forwarded_entry_cve   assess=attack  contain=[checkout-api-...]   ← NEW fixture: route-forwarded backend promotes via the entry lane ✓
[OK] downstream_cve_only         no_attack / []     ← JEF-588 trap: no drift ✓
[OK] downstream_behavioral       attack / {web-cache-0}   ← minimality centerpiece: no drift ✓

Acceptance met: the new fixture scores attack + correct cut-set on the deployed judge, and the two named drift checks are unchanged.

Note (pre-existing, out of scope for this PR): downstream_secret intermittently returned UNPARSEABLE on the deployed 1.7b (a re-run at temp-0 gave 1/3 attack, 2/3 unparseable — CPU/backend non-determinism producing malformed JSON on the exposed-secret evidence shape). It degrades to the skeptic default (safe, no false cut) and is not affected by this fixture-only change. Tracked separately.

@thejefflarson
thejefflarson merged commit 7f528f4 into main Aug 1, 2026
5 checks passed
@thejefflarson
thejefflarson deleted the thejefflarson/jef-698-deployed-pod-bench-route-forwarded-backend-fixture branch August 1, 2026 21:36
thejefflarson added a commit that referenced this pull request Aug 1, 2026
…, drop stale exemption

The breadcrumb sweep mangled two `// JEF-317, ON-NODE PENDING` trailing
comments in agent/protector-agent-ebpf/src/vmlinux.rs into `/, ON-NODE
PENDING` inside a `const _: () = { ... }` compile-time offset guard —
invalid Rust that would break the ebpf cross-compile. Restore the `//`
markers (JEF-317 dropped per the sweep).

#314 (which was concurrently editing scripts/judge_bakeoff_cutchoice.py)
is now merged, so its temporary EXEMPT_FILES entry is stale. Sweep the 3
remaining breadcrumbs there (keep the CUT TRAP / REFUTE TRAP mechanism
names; ADR-0034 is already cited inline) and remove the exemption, leaving
the guard with a single structural exemption (its own source file).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP
thejefflarson added a commit that referenced this pull request Aug 1, 2026
… sweep breadcrumbs (#316)

* ci(guard): enforce the committed-content self-containment guardrail + sweep breadcrumbs

Adds engine/tests/self_containment_guard.rs, a repo-wide nextest guard (alongside
file_size_guard.rs) that fails the build on a JEF-nnn ticket reference or a
linear.app URL in tracked source comments/docstrings and docs, with narrow,
documented exemptions for the guardrail's own naming text in CLAUDE.md and for
scripts/judge_bakeoff_cutchoice.py (mid-edit on a concurrent branch).

Sweeps every existing JEF-nnn/linear.app breadcrumb the guard would flag out of
tracked source and docs, replacing each with the ADR it records, the module it
lives beside, or the inline reasoning already next to it — preserving the
rationale, never the ticket number. Comment/doc-prose only; no code-logic change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP

* ci(guard): fix mangled ebpf comment marker + fold judge_bakeoff sweep, drop stale exemption

The breadcrumb sweep mangled two `// JEF-317, ON-NODE PENDING` trailing
comments in agent/protector-agent-ebpf/src/vmlinux.rs into `/, ON-NODE
PENDING` inside a `const _: () = { ... }` compile-time offset guard —
invalid Rust that would break the ebpf cross-compile. Restore the `//`
markers (JEF-317 dropped per the sweep).

#314 (which was concurrently editing scripts/judge_bakeoff_cutchoice.py)
is now merged, so its temporary EXEMPT_FILES entry is stale. Sweep the 3
remaining breadcrumbs there (keep the CUT TRAP / REFUTE TRAP mechanism
names; ADR-0034 is already cited inline) and remove the exemption, leaving
the guard with a single structural exemption (its own source file).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VtjoJttCvBY4dzCoE4f9vP

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant