feat(proxy): add observe mode — measure what compaction would save, without enforcing it

[OsherElhadad]

Partially closes #31: observe mode lands here; async is deferred to #35 (kept as a
draft, all its review fixes intact).

Why split

#31 specified three modes. Observe is the smaller, lower-risk half and delivers the
adoption story on its own — the issue itself suggested landing it first. Async is held
because its measured benefit has since evaporated:

• its headline latency win (1,599 ms → 25 ms) came almost entirely from deferring
  extract_llm, and perf(extract_llm): economic gate, global result cache, and the honest verdict (#28) #34 hard-declines extract_llm on caching backends — so on the
  primary workload there is nothing expensive left to defer. Terminal-Bench already showed
  exactly this: zero LLM calls, async 26.8 ms vs sync 26.9 ms;
• its cache-write evidence was withdrawn under review (the tail protection turned out to be
  inert on claude-code, which sets its own breakpoints), so re-establishing it needs a fresh
  50-task paired arm;
• its dropped / stale_discarded paths — the ones guarding the correctness invariant —
  are exercised by tests only, never by production load.

Carrying a bounded worker pool, per-session generations, a copy-on-write store overlay and a
cache policy that must decline on the default agent is a lot of concurrency surface for an
unproven benefit. The primitives held up under a hostile review, which is why #35 stays on a
branch rather than being discarded: it is ready to measure, not ready to merge.

What observe does

The agent receives its request untouched, byte for byte.

Byte-identity is structural, not a property of careful copying: the request path does
not run the pipeline at all, and skips expand.Inject too — injecting a tool declaration
would be a modification. There is no code path in observe mode that could alter a forwarded
body. A test asserts it anyway, including on a tool-carrying request where expand.Inject
would otherwise append.

A copy runs off-path against observe's own store, purely to record what compaction would
have achieved.

mode: observe     # sync (default) | observe

Cost to the enforced path: 0.062 ms/req, against sync's 1,599 ms on the same
benchmark. Confirmed independently in a live Claude Code session (0.209 ms vs 28.964 ms) and
on Terminal-Bench (0.076 ms vs 26.9 ms).

headroom has no observe/shadow/dry-run mode at all — its token and cache modes are both
enforcing, and its only control arm is a 10% output-shaper holdout. This is a genuine
differentiator, not a port.

Why the projection is trustworthy

Both requirements were found by comparing the projection against sync's actuals, not by
reading code — and each was a real bug:

• It shares the cached-prefix boundary the enforced path uses. Without that,
  cache-awareness gates nothing, every message looks compactable, and the projection
  overstates by exactly what cache-awareness costs: 9.5% projected against 0.8% actually
  achieved on the same tasks. A projection that ignores cache-awareness is a projection of
  what a cache-blind proxy would do.
• Its store is persistent. Offloaders freeze a decision and replay it on every later
  turn, which is where most of the sustained saving lives. Against a discarded buffer observe
  sees only the current tail and under-projects by 3x.

It is nevertheless disjoint from the live store: a decision observe made must never be
replayable by a real request — that would be a request modification arriving by the back
door. A test asserts zero writes to the live store.

Three lines of evidence, strongest first:

Evidence	Result
Controlled same-traffic comparison	exact: 10,020 tokens / 23.06% both sides
Terminal-Bench (negative control)	sync 1.02% actual → observe 0% projected
SWE-bench arms	6.40% vs 0.82% — consistent but noisy, reasons given, not explained away

The negative control is the more convincing shape: on traffic with almost nothing to save,
observe reports almost nothing rather than inventing a headline. A mode that only ever
agreed on high-savings traffic would be far weaker proof.

Metric namespace separation

Observe results live in physically separate accumulators serialized under potential_*
/ projected_*, sharing no key with an enforced metric. In observe mode every enforced
savings aggregate is zero by construction — requests, saved_tokens, sync_enforced,
components: {} — which is the machine-readable form of "context-guru did not modify
requests". A mislabelled hypothetical silently inflates the headline claim, so this is a
correctness boundary, and a test asserts no enforced aggregate can reach an observe result.

Two enforced keys are deliberately real there rather than zeroed:

• cg_added_ms_avg — the actual enforced-path latency. Reading ~0 is the headline;
  zeroing it would hide the result.
• llm_calls / llm_input_tokens / llm_output_tokens — context-guru's own model spend.
  Observe measures off-path and that costs real money, so relabelling it potential_* would
  be a worse lie than leaving it. It stays where cost tooling reads it, labelled by
  observe_llm_notice as the cost of measuring rather than enforcing.

Also included: the prevLen race fix

Independent of modes, and worth flagging for #40's author. The per-session cached-prefix
boundary was read from the store and written back in a defer, so two concurrent turns of
one session both read the same length and the second's write-back could land first — leaving
a boundary that describes neither turn. A boundary that is too high lets an offloader mutate
content the provider has already cached, costing a full cache-write of the suffix.

modes.Tracker.Turn now reads and records it in one locked call. Callers without a tracker
(library users, /compact) keep the legacy path unchanged.

store.Buffer / FrozenLoser — async-only, flagged deliberately

I raised that a component asserting c.Store.(store.FrozenLoser) would see a wrapper and
silently take the degraded path. That concern does not apply to this branch: observe's
shadow store is a plain *store.Memory, which implements FrozenLost natively. The
copy-on-write store.Buffer exists only on #35, where the forwarding fix lives with it.
Nothing is lost by #35 going to draft.

Compatibility

mode: sync remains the default and is byte-identical to today, asserted by a golden test
comparing the two entry points' output. /stats gains mode, sync_enforced and the
observe hypotheticals; every pre-existing field keeps its name and shape (a test pins the
full set, since deploy/harbor/*.py parses this payload). /compact is mode-independent by
contract, also tested.

Rebased onto current main, so it includes #36 and #40; the projection-agreement result was
re-verified after that rebase.

Tests

go test -race throughout:

• sync byte-identical to the legacy entry point, and unset mode identical to explicit
  sync (golden);
• observe leaves the forwarded body byte-identical, including a tool-carrying request that
  expand.Inject would otherwise modify;
• observe's projection agrees with sync's actual on identical traffic (fails at ratio 0.33
  without the shadow store);
• observe never writes the live store;
• observe metrics unreachable from every enforced savings aggregate, and the two
  vocabularies disjoint in the serialized payload;
• concurrent turns of one session cannot corrupt the boundary, and no concurrent write is
  lost;
• enqueue dedup atomic against 32 concurrent enqueues of one key; a full queue drops, counts
  and never blocks; a panicking observation is contained; Stop does not wait on an
  uncancellable job; no goroutine leak on Stop or Handler.Close, both idempotent;
• /stats backward compatibility; /compact mode-independence; unknown mode rejected
  (including async, which is not valid on this branch).

Gates: build · test -tags cg_skeleton · -race -count=5 · gofmt -l · go vet ·
mkdocs build --strict — all clean.

Docs

New docs/how-to/operating-modes.md (when to use each, how to read the projection, and
what observe cannot tell you — cache effects are projected not measured, no expand bounce
is exercised, and measuring still spends cheap-model tokens). New
docs/results/observe-mode.md with the three evidence lines and a "what is not established"
section. docs/design.md, docs/reference/config.md, docs/how-to/measure-savings.md,
mkdocs.yml, README.

Remaining for #31

Async mode, on #35 (draft): benefit conditional and currently unmeasured now that
extract_llm no longer runs on caching backends. What would establish it: a 50-task paired
arm with async.strip_caller_breakpoints: true, showing cache-write no higher than sync's
and a real latency reduction on traffic that still makes model calls.

[OsherElhadad]

Rebased onto current main (now includes #36, #40 and #42). Three conflicts resolved, all pure unions — #42's cmdfilter ledger fields and my mode fields are additive in the same Aggregator/Snapshot regions, and #40's FrozenLoser work needed no adjustment here.

Re-verified after the rebase: the projection-agreement result is unchanged (sync saved 10,020 / 23.06%, observe projected 10,020 / 23.06% — exact), and all gates are green including -race -count=5.

Worth restating since #40 has now landed: observe's shadow store is a plain *store.Memory, so it picks up FrozenLost natively and there is no wrapper to forward through. The store.Buffer → FrozenLost forwarding I flagged belongs only to the async branch (#35, draft), where store.Buffer lives.