feat: server-side availability oracle in the operator path (DD-030)

[basquin-bot]

What & why

The operator path had no server-side availability oracle. The operator injected -javaagent:basquin-agent.jar, but Agent.premain was a no-op stub and no valve was mounted — so Agent.beginIteration()/endIteration() were never called in the target JVM. In-cluster explore campaigns got real coverage (JaCoCo TCP from the target) but their heap/thread/latency findings came from the driver JVM (client-side), not the app. The project's core thesis — heap retention and thread/executor leaks inside the app under test — was inert in Kubernetes.

This PR brings the server-side oracle (and DD-029 lock-free load) to the operator path.

How

The agent installs the request boundary itself via bytecode — no valve mount, no context.xml/lib/ surgery, works on any Tomcat image:

• agent/RequestBoundary.java (new) — the three-state boundary (/__basquin control · load passthrough · explore lock+begin/end), extracted from the valve. Catalina-free (agent classes load on the boot loader, which can't see Catalina), so it's String-in / decision-out and unit-tested with no fakes.
• agent/TomcatBoundaryAdvice.java + Agent.premain (new) — a ByteBuddy advice inlined into org.apache.catalina.core.StandardHostValve.invoke, installed only when -Dbasquin.boundary=agent (default off).
• BasquinValve — refactored to delegate to RequestBoundary (bench/manual path; behavior preserved, namespace-free per DD-011).
• operator — appends -Dbasquin.boundary=agent (it mounts no valve, so it opts in). One line.
• e2e — asserts /__basquin/drift (CSV) + mode control are live on the valve-less operator target — closing the loop the removed DD-029 assertion left open.

Opt-in default-off matters: the bench path runs the valve and the agent; if the agent boundary were always on it would double-count. The operator opts in; the bench path doesn't.

⚠️ Operational notes for adopters

1. Rolling re-injection on operator upgrade. -Dbasquin.boundary=agent is part of buildAgentArgs, which feeds specHash, so every already-injected BasquinTarget re-injects once on upgrade → a one-time rolling restart of instrumented Deployments. Expected and correct.
2. Operator explore targets now serialize. For the first time, operator explore-mode requests go through ITERATION_LOCK (concurrency capped at 1) — the intended cost of trustworthy per-request server-side deltas (DD-005/010). Load mode (DD-029) stays lock-free.

Verification

• Unit: RequestBoundary state machine (5 tests) + full JVM suite (95/95); valve stays namespace-free (javap = 0); agent jar bundles ByteBuddy, not Catalina.
• Operator: controller suite asserts the injected flag.
• The in-cluster e2e is the integration proof that premain instruments a real Tomcat and serves /__basquin from a valve-less operator target. (Known watch-item: the ByteBuddy ClassFileLocator finding the boot-classpath advice bytes at premain is proven only by this e2e; a documented ofSystemLoader() fallback exists if it ever no-ops.)

Depends on

The merged DD-029 work (#70) — LoadMode / LoadModeControl.

Spec: docs/superpowers/specs/2026-07-21-operator-server-side-boundary-design.md
Plan: docs/superpowers/plans/2026-07-21-operator-server-side-boundary.md

🤖 Generated with Claude Code

[basquin-bot]

@claude please review — server-side oracle in the operator path via agent bytecode (DD-030). Note the two operational heads-ups in the description (rolling re-injection on upgrade; explore targets now serialize).

[ianp94]

The design here is right — shared RequestBoundary with correct lock discipline (lock-before-begin, unlock-and-degrade on begin failure, headers computed before unlock), exact preservation of the valve's exception semantics, and the default-off opt-in that keeps the bench path single-boundary. The unit tests cover the state machine well.

But the in-cluster e2e — which this PR itself names as the integration proof — fails, and it exposes two real defects:

1. The driver JVM now crashes on NoClassDefFoundError: net/bytebuddy/matcher/ElementMatcher

Exception in thread "main" java.lang.NoClassDefFoundError: net/bytebuddy/matcher/ElementMatcher
	at runner.coverage.CoverageGuidedRun.runSequence(CoverageGuidedRun.java:377)

The new ByteBuddy imports in Agent.premain mean that linking agent.Agent now requires ByteBuddy on the classpath (the verifier resolves the premain method's referenced types when the class links). The driver/runner bundles the agent classes but not ByteBuddy, so the first driver call into Agent kills the campaign — phase=Failed, coveragePct=<none>, no corpus ConfigMap; every campaign-stage e2e failure cascades from this one error.

Suggested fix: move the ByteBuddy installation into a separate class (e.g. agent.BoundaryInstaller) and invoke it reflectively from inside the flag check in premain (Class.forName("agent.BoundaryInstaller")...). Then Agent itself never references ByteBuddy types, the driver links it exactly as before, and a missing-ByteBuddy environment degrades gracefully inside the existing try/catch. Bundling ByteBuddy into the runner jar would also work but is heavier and fixes only this occurrence, not the coupling.

2. The agent boundary never installed on the target — /__basquin/* returns Tomcat's 404 page

drift=<!doctype html>...HTTP Status 404 – Not Found...Apache Tomcat/9.0.120

The app is healthy (200s, agents loaded, coverage endpoint up), so premain ran — but StandardHostValve was never instrumented. This is precisely the watch-item the PR description flags: Advice.to(TomcatBoundaryAdvice.class) failing to locate the advice class-file bytes for a class on the boot classpath. The e2e has now demonstrated the default lookup does not work in exactly the environment this PR exists to serve, so the documented ofSystemLoader() fallback needs to be the implementation, not a contingency — e.g.:

Advice.to(TypeDescription.ForLoadedType.of(TomcatBoundaryAdvice.class),
          ClassFileLocator.ForClassLoader.ofSystemLoader())

Please also grab the target pod's catalina stdout on the next e2e run to confirm which premain message fired (installed vs NOT installed: ...) — if it's the latter, the exception text will confirm the locator diagnosis; if premain printed installed yet requests still bypass the boundary, that's a different bug (matcher never firing) and worth knowing before re-pushing.

Minor, non-blocking

• BasquinValve.invoke returns early on the control path without calling RequestBoundary.onExit, while the advice path does call it. Benign today (every onEnter path resets the thread-local), but calling onExit in both keeps the enter/exit contract uniform and the thread-local clean between pooled-thread requests.
• Unshaded ByteBuddy on the boot classpath will shadow any app-bundled ByteBuddy version (parent-first delegation). Pre-existing, but worth a backlog note now that premain actually exercises it.

Happy to approve once the e2e is green — the two fixes above are both small and well-localized.

[ianp94]

Re-reviewed after e9be5f5. Issue 1 is fixed and verified — the campaign now completes with non-zero coverage and an emitted corpus ConfigMap, so the reflective BoundaryInstaller isolation works exactly as intended. The valve header-write guard and doc cleanups look good too.

Issue 2 is still present: the re-run e2e still gets Tomcat's own 404 for /__basquin/drift and /__basquin/mode (Tomcat 9.0.120), so StandardHostValve is still not instrumented — the explicit ClassFileLocator didn't resolve it.

The blocker for diagnosing further is that the e2e failure dump never shows the target pod's Basquin startup lines. The job log contains app exception stacks but no Basquin Agent initialized / [Basquin] agent boundary installed / [Basquin] agent boundary NOT installed: lines, so we can't distinguish the three candidate failure modes:

1. premain never saw the flag (injection/env plumbing),
2. install() threw (the NOT installed: message + cause would name it),
3. installOn succeeded but the transform never fired (the errors-only listener output, or a matcher/TypePool problem — e.g. the advice's Catalina parameter types resolving against a locator that can't see catalina.jar, which lives in Tomcat's common loader, not the system classpath).

Please add to e2e.sh, next to the DD-030 checks (or in the failure path):

$K -n "$NS" logs "$apod" -c jpetstore --tail=400 2>/dev/null | grep -iE 'basquin|bytebuddy' | sed 's/^/    /' || true

so the next run pinpoints which path fired. Worth also reproducing locally before the next push — docker run the jpetstore image with the agent jar, -Xbootclasspath/a, and -Dbasquin.boundary=agent, then curl /__basquin/drift — since CI is currently the only integration signal and each round trip is slow.

If it turns out to be failure mode 3 (Catalina types unresolvable from the advice's pool at transform time), consider making the advice Catalina-free like RequestBoundary itself: take the arguments as Object (@Advice.Argument(value=0, typing = Assigner.Typing.DYNAMIC)) and do the four Request/Response calls via cached MethodHandles. That removes the compileOnly tomcat-catalina dependency and every cross-loader resolution concern at once, at the cost of a slightly uglier advice body.

Keeping my request-changes until the DD-030 e2e assertions are green.

[ianp94]

Approving — all checks green, including the in-cluster e2e, whose DD-030 assertions now prove the whole chain: premain → reflective BoundaryInstaller → target-loader TypePool resolution → instrumented StandardHostValve serving /__basquin/drift (real CSV: heap/threads/epoch) and /__basquin/mode (ok:explore) on a valve-less operator target.

The three review rounds landed exactly the right fixes:

1. Driver decoupled from ByteBuddy (BoundaryInstaller reached only reflectively) — campaign runs are healthy again.
2. Advice resolved via the target class's loader — the cross-loader silent no-op is gone, and the errors-only listener ensures any future transform failure is loud.
3. Valve header-write guard + doc fixes from the earlier nits.

Two small follow-ups worth a backlog note (non-blocking):

• Add the target-pod grep -iE 'basquin|bytebuddy' log dump to e2e.sh's DD-030 failure path — this round trip would have been one run shorter with it.
• Unshaded ByteBuddy on the boot classpath will shadow any app-bundled ByteBuddy (parent-first); now that premain exercises it, shading/relocation deserves a ticket.

Nice work on the turnaround — the final design (Catalina-free RequestBoundary, glue-only advice/valve, opt-in flag) is clean and well-documented.