v0.10: coverage over HTTP, grammar-driven exploration, k8s demo, and a standalone dashboard

[ianp94]

Summary

Started as "coverage over HTTP" and grew into the full v0.10 milestone: the harness can now measure the app under test's own coverage, use it to guide exploration, run against unmodified third-party apps in Kubernetes, and report findings through a standalone dashboard. Reasoning for every significant choice is in docs/DESIGN-DECISIONS.md (DD-012 … DD-022).

What it does now, end to end

Pointed at an unmodified JPetStore running as a pod in kind:

Coverage of the app's own code	23.1% (1469/6368 probes), read from its JaCoCo agent over the wire
Distinct crash sites found in the app	7
Findings clustered from	937 raw → 19 unique

Coverage progression, each step a real fix rather than tuning: 4.4% (round-robin) → 8.6% (guided mutation) → 17.3% (routes as a corpus, not hardcoded) → 17.7% (structural value generation) → 22.1% (sessions) → 23.1% (multi-step transactions).

Real defects found in the app under test

NullPointerException: "accountBean" is null
  at OrderActionBean.listOrders(OrderActionBean.java:110)

listOrders, viewOrder, newAccount, editAccount, viewItem and addItemToCart dereference lookups that can legitimately return null, so an unauthenticated or not-found request returns 500 instead of a redirect or a 404. OrderActionBean also reads the session account bean under two different keys with different lifecycles (/actions/Account.action vs accountBean), neither null-checked.

Main pieces

• Coverage over HTTP (DD-012) — JaCoCo agent in the app JVM; the driver dumps and analyses it client-side. Coverage-guided mutation keeps inputs that reach new code.
• Exploration surface as data (DD-016/017/018) — a request grammar supplies route templates, the corpus supplies values, and ~EST-[0-9]{1,4} structural generation invents ids that parse but don't exist. @sequence blocks run ordered session-carrying transactions; placeholders bind once per execution so a transaction is coherent.
• Sessions (DD-018) — JSESSIONID carried across steps, alternating authenticated and anonymous epochs.
• Kubernetes demo — one command brings up JPetStore as a pod with valve + agent + JaCoCo baked in.
• Standalone dashboard (DD-013) — drivers push; the dashboard never drives anything. Fleet view, run configuration (params + grammar), findings clustered by fingerprint with the concrete inputs that produced them, and optional Claude-backed analysis.
• Findings are triageable (DD-014/019/020) — clustering is read-path only so the corpus stays whole; crash findings carry the app's stack, parsed from the error response, not the harness's.
• Tests (DD-021) — 41 tests, 31 new, covering the parsing/generation logic that previously had none. Verified by mutation, not just by passing.

Review fixes (all four blockers)

• Dashboard security (DD-022) — was all-interfaces + Access-Control-Allow-Origin: * + an unauthenticated endpoint that spends API credit. Now binds 127.0.0.1, sends no CORS headers, and requires a guard header on ingest/analyze (a cross-origin simple request can't set one, and the preflight fails). Optional shared token; analyze refuses on a non-loopback bind without one. Verified: listener is loopback-only, both endpoints refuse without the header, no Access-Control-* on any response.
• Wrong-cluster deploy — every kubectl call pinned to --context kind-$CLUSTER.
• Exposed JaCoCo port — Service is ClusterIP; its remote-control protocol is unauthenticated and can reset execution data.
• Silently stale image — unique tag per build plus an explicit set image.

Also from the review: Claude requests disable thinking and report a max_tokens stop; coverage caches class bytes instead of re-parsing every class per iteration; COVERAGE_INCLUDES defaults to the app package; compose uses stable staged paths; up.sh/gradlew executable; dashboard has checked fetches, a real liveness indicator and a generation counter against racing polls; plus the smaller escaping/parsing items.

Known limits (documented, not fixed here)

• One driver behind a Service with N replicas under-reports coverage ~1/N — JaCoCo's connection lands on one pod while requests load-balance across all. Needs a merge across replicas.
• Sessions need sessionAffinity: ClientIP in that topology; findings don't record which replica served them.
• The Claude path is verified only for request shape and error handling, not against a live key.
• No POST/form-body support yet.

Docs

docs/USAGE.md now documents every v0.10 flag, task, and the grammar format; deploy/k8s/README.md covers the ClusterIP and context decisions; README describes coverage as shipped rather than upcoming.

🤖 Generated with Claude Code

[ianp94]

Review — build is green, but several deployment/security issues should be fixed before this merges

Verified locally

Built and tested the branch at d4a457a in an isolated worktree: gradle build test passes, 41 tests across 11 suites, 0 failures (up from 10 tests on main). The DD-021 test work is real — RequestGrammarTest (14), FindingsClustererTest (10), and JsonScanTest (7) cover the parsing/generation logic that previously had none.

I also confirmed the JsonScan stack-overflow rationale is sound, and empirically checked that ClaudeAnalyzer.extractText still returns the right value when the response contains thinking blocks ahead of the text block (it does — the "type":"text" match happens to land on the following "text": key).

Not verified: the kind/JPetStore in-cluster claims, the 96 invariant finds, and the 281/6368 coverage figure — no cluster available here.

Findings, most severe first

1. The dashboard server is unauthenticated, binds all interfaces, and exposes an endpoint that spends money. DashboardServer.main binds new InetSocketAddress(port) (all interfaces) with no auth, and respond() sets Access-Control-Allow-Origin: * on every response. POST /api/analyze/{id} calls the Claude API using the operator's key. A plain cross-origin POST with no custom headers isn't preflighted, so any page the operator visits while the dashboard is running can trigger billed API calls in a loop — and anyone on the same network can do so directly. At minimum: bind to loopback by default, drop the wildcard CORS header, and require a shared token on /api/analyze/.

2. deploy/k8s/up.sh never targets the kind cluster. kind create cluster is what sets kubectl's current-context, and line 32 skips it when the cluster already exists. Lines 38–39 then kubectl apply and kubectl rollout status against whatever context is currently active. A developer whose current-context points at a shared or staging cluster, who has previously created the kind cluster, gets a Deployment plus a NodePort Service in the remote cluster on re-run. kind load on line 35 is correctly scoped with --name, which makes the gap easy to miss. The same applies to the printed port-forward hint and the README's teardown command. Fix: --context "kind-$CLUSTER" on every kubectl invocation.

3. The JaCoCo tcpserver is published as a NodePort. jpetstore.yaml:41-42 exposes port 6300 as nodePort: 30063, and Dockerfile.jpetstore:13 binds it to 0.0.0.0. JaCoCo's remote-control protocol has no authentication and lets any client both dump and reset execution data. The file's own header comment and the README both say access is via kubectl port-forward, which only needs ClusterIP — so the NodePort isn't buying anything. Switch the Service to ClusterIP and drop both nodePort entries.

4. Re-running up.sh silently keeps the old image while reporting success. The tag is fixed at closurejvm/jpetstore-demo:latest and the manifest is byte-identical between runs, so kubectl apply is a no-op, no new ReplicaSet is created, and rollout status returns "successfully rolled out" immediately. With imagePullPolicy: IfNotPresent, the pod keeps the previously loaded image. Edit the agent, re-run, and you measure the old build while being told it deployed. Needs kubectl rollout restart deploy/jpetstore or a unique tag per build.

5. ClaudeAnalyzer will truncate its own output on the model it defaults to. It sends max_tokens: 800 with no thinking parameter, and the default model is claude-sonnet-5 — which runs adaptive thinking when thinking is omitted. Thinking tokens count against max_tokens, so an 800-token ceiling can be largely or entirely consumed by thinking, leaving a truncated analysis or none at all. There's also no stop_reason check, so a max_tokens stop is indistinguishable from a complete answer. Either set "thinking":{"type":"disabled"} for this summarization task, or raise max_tokens substantially and check stop_reason. (Model ID, endpoint, x-api-key / anthropic-version: 2023-06-01 headers, and request shape are all correct.)

6. Coverage sampling re-reads and re-parses every class file from disk on every iteration. JacocoCoverageProvider.sample() walks classFiles and calls Files.newInputStream(...) + analyzer.analyzeClass(...) for each one, and CoverageGuidedRun calls sample() once per iteration. The class files never change during a run, so this is O(classes) disk I/O and bytecode parsing per HTTP request — for JPetStore that's hundreds of files re-read hundreds of times. Caching the class bytes in memory at construction time would make the sampling cost a socket round-trip plus in-memory analysis.

7. deploy/k8s/up.sh is committed non-executable (mode 100644 — the only .sh in the repo). Both the README and the script's own header document JPETSTORE_WAR=... deploy/k8s/up.sh, which fails with Permission denied. gradlew is also 100644 (pre-existing on main), so even bash deploy/k8s/up.sh dies at the first build step. Needs git update-index --chmod=+x on both. Related: the CRLF line endings on gradlew made sh gradlew fail outright in a fresh worktree — I had to use standalone Gradle.

8. COVERAGE_INCLUDES default contradicts the comment above it. docker-compose.coverage.yml:24 has includes=${COVERAGE_INCLUDES:-*} directly under a comment saying "Scope JaCoCo's instrumentation to the app package … so we measure the app, not libs." Nothing sets the variable and there's no .env, so the documented invocation instruments Tomcat, MyBatis, and the whole servlet stack. That inflates the coverage denominator (making the dashboard's "% explored" wrong) and adds enough overhead to produce false positives against the -Dclosurejvm.invariant.latency.maxMs=25 threshold on line 13. Dockerfile.jpetstore gets this right with a hardcoded org.mybatis.jpetstore.* — the compose default should match.

9. The dashboard silently swallows every fetch failure while showing a green "live" indicator. All API calls are await (await fetch(x)).json() with no res.ok check, wrapped in empty catch(e){} blocks. Separately, the header dot #hdot is never touched by any code — the .dot.dead style exists and is applied to per-campaign pods, but the header dot is permanently green. If the server dies or an endpoint 500s, the page shows "live" with frozen counters indefinitely and the operator reads stale numbers as current.

10. Overlapping polls can overwrite fresh data with stale. setInterval(..., 1500) plus direct tick() calls from three click handlers, where tick() awaits two sequential fetches. When the server is slow — plausible while /api/analyze is running, since it shares the same HttpServer — several tick()s are in flight and last-to-resolve wins. The pod-click handler also races an in-flight tickFleet() that will clobber the subtitle back to fleet view. A generation counter or AbortController would fix both.

Smaller items

c.classification is the one server field interpolated unescaped, and it goes into a class attribute; esc() only handles & < >, not quotes, so it wouldn't be sufficient there anyway. Not currently exploitable — the values are harness literals — but it's read back unvalidated from on-disk meta files. Worth noting the genuinely target-controlled data (kind, from the app's response header) is correctly escaped, so the real XSS path is closed.

DashboardServer.queryParam reads getRawQuery() without URL-decoding, while campaignDetail splits getRequestURI().getPath(), which is decoded — so a campaign id needing any encoding is stored under one key and looked up under another. Pod names are DNS-safe, so this won't bite in practice today.

CoverageGuidedRun parses closurejvm.coverage.jacoco with substring(0, indexOf(':')), which throws a confusing StringIndexOutOfBoundsException if the value has no colon. total is only assigned on the single-request path, so a run consisting entirely of sequences prints coverage=N/0. And ClaudeAnalyzer carries a dead private unescape() method (lines 109–127) duplicating the one in JsonScan.

Hardcoded artifact versions in docker-compose.coverage.yml:27-28 (closurejvm-0.2.0.jar, closurejvm-valve-0.1.0.jar) match today's build files, but on a version bump Docker creates the bind-mount source as an empty directory, so Tomcat boots with -javaagent: pointing at a directory and aborts with an unrelated-looking error. The copyJacocoAgent task already shows the better pattern — a stable, version-independent output path.

Finally: the PR description covers coverage-over-HTTP and the kind demo, but the branch has grown to 13 commits including the web dashboard, grammar-driven fuzzing, findings clustering, the Claude analyzer, multi-step transactions, and the new test suite. Worth refreshing the body before merge so reviewers know what they're looking at.

Overall

The engineering is strong and the design-decision log makes the reasoning easy to follow. Two things I specifically tried to disprove and couldn't: the rename '.*', 'jacocoagent.jar' copy task does not double-apply, and testImplementation sourceSets.coverage.output is safe despite not pulling org.jacoco.core. The JsonScan extraction is also genuinely correct against realistic Claude response shapes, including with thinking blocks present.

My merge blockers are #1–#4 — the unauthenticated paid endpoint, the wrong-cluster deploy, the exposed JaCoCo port, and the silently-stale image. The rest can follow up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UH8Wf39y1RFfcTKqf3d7uF