Feature/0.0.1

.gitignore

@@ -35,3 +35,14 @@ go.work.sum
 /trace.jsonl
 /report.md
 /report.json
+/model.json
+/predicted-trace.jsonl
+/cloudoracle-trace.jsonl
+/cloudoracle-model.json
+
+# Python sidecar artifacts
+__pycache__/
+*.py[cod]
+.pytest_cache/
+*.egg-info/
+.venv/

README.md

@@ -182,6 +182,34 @@ Retries and fan-out scale the whole call (more calls); context growth inflates
 only the prompt side, not the completion — so the model reflects *which* driver
 moved, not a flat fudge factor.
 
+### Predict cost without running (Python sidecar)
+
+The what-if knobs above re-cost a recorded trace, but they hold output length
+fixed. The one thing you genuinely can't guess for an agent is **completion
+length** — and it's where the cost surprise lives. The optional
+[`sidecar/`](sidecar/) (Python) learns output length from a recorded trace, then
+projects cost for inputs you never ran:
+
+```sh
+# Learn output_tokens ≈ a + b·input_tokens per model, from one recorded run.
+python -m augur_predict fit --trace trace.jsonl --out model.json
+
+# "What if prompts grow 1.5× as conversations lengthen?" — predict, don't run.
+python -m augur_predict emit-trace --model model.json --out grown.jsonl --input-scale 1.5
+
+# The predicted trace flows through the normal gate — no tokens spent.
+augur gate --trace grown.jsonl --traffic traffic.yaml --budget budget.yaml
+```
+
+Coupling is the trace file and nothing else — no RPC, no shared library. Unlike
+`--context-growth`, the sidecar feeds the larger prompt *back through the output
+model*, so a bigger ask predicts a longer answer, not just a costlier prompt.
+The default fit is an honest linear baseline (it reports R² and falls back to the
+mean when the signal is weak); `--dist quantile` fits conditional quantiles
+directly to model the right-skewed tail the gate's p95 lives in, and
+`--run-correlation` widens the per-run spread to match correlated runs. See
+[`sidecar/README.md`](sidecar/README.md).
+
 ### Self-hosted models (TCO)
 
 For a model you run yourself there's no per-token API price — you pay for an
@@ -252,7 +280,8 @@ example.
   truthfully. Classifying those calls into retries vs sub-agent fan-out needs
   labeling the trace does not yet carry.
 - **Running the agent in CI spends real tokens.** Keep the scenario set small and
-  `runs` modest; a record-once/replay mode is on the roadmap.
+  `runs` modest, or record once and replay (`--record`/`--replay`) so CI pushes
+  spend nothing.
 
 ## Status
 
@@ -266,13 +295,42 @@ dependency is `gopkg.in/yaml.v3`):
 | Hito 2 | scenario runner + per-scenario aggregation |
 | Hito 3 | projection engine with bootstrap confidence intervals |
 | Hito 4 | budget gate + Markdown/JSON report + CI exit codes |
-| Hito 5 | record/replay cassette (`--record`/`--replay`), what-if knobs (`--retry-rate`/`--fanout`/`--context-growth`), self-hosted TCO mode (`augur tco`, `--tco`), GitHub Action (`action.yml`) |
+| Hito 5 | record/replay cassette (`--record`/`--replay`), what-if knobs (`--retry-rate`/`--fanout`/`--context-growth`), self-hosted TCO mode (`augur tco`, `--tco`), GitHub Action (`action.yml`), Python output-length prediction sidecar ([`sidecar/`](sidecar/)) |
 
-**Roadmap (stretch):** a Python output-length prediction sidecar (the analytical
-piece that would earn a second language).
+Every SPEC milestone and stretch is implemented. The Go core is pure Go (only
+external dependency `gopkg.in/yaml.v3`); the optional prediction sidecar is
+Python (numpy + scipy), coupled to the core through the trace file alone.
 
 See [`SPEC.md`](SPEC.md) for the full design.
 
+## Dogfooded on a real agent
+
+Augur has been run against a real, non-trivial agent — the Insights Agent from
+its sibling project [**CloudOracle**](https://github.com/Cro22/CloudOracle) (a
+LangGraph supervisor multi-agent with tool calls and guardrails), not just
+synthetic traces. CloudOracle is FinOps for *cloud* (runtime); Augur is FinOps
+for *AI agents* (pre-prod) — dogfooding one on the other closes the loop.
+
+Because that agent talks to its model natively through LangChain (no OpenAI
+`base_url`), the capture used the SPEC's documented proxy fallback (ADR D1): a
+LangChain **callback shim** that writes Augur's trace schema from the usage every
+call reports — *without editing CloudOracle*. Full walkthrough and harness:
+[`examples/cloudoracle/`](examples/cloudoracle/).
+
+**What it found (Claude Haiku 4.5, 20 runs): gate PASS** at `$/request p95
+$0.0198` (budget $0.02). The headline is the `find-savings` scenario — its **p95
+cost is 2.3× its median**, driven by a call-count tail (5 → 13 calls/run when the
+savings specialist's tool loop keeps going). That agentic cost driver, observed on
+a real agent, is exactly what Augur exists to catch; gating on the mean would hide
+it.
+
+Dogfooding also surfaced real findings *about the agent* — it isn't
+provider-portable (its LLM judge sends a system-only message Anthropic rejects but
+Gemini tolerates), Claude Haiku hallucinates dollar figures that trip the agent's
+grounding check, and the sidecar's input→output signal is weak there (output
+length is driven by call *role*, not prompt size). Details in the
+[example README](examples/cloudoracle/README.md).
+
 ## Naming
 
 **Augur** — a seer who reads omens to foretell what's coming; here, an agent's

SPEC.md

@@ -177,6 +177,7 @@ budget.yaml ──┐││                                  LLM calls (base_url
 ## 10. Open questions for Jesús (decide before/at Hito 0)
 
 1. **First dogfood target.** Strong candidate: **CloudOracle's own Insights Agent** (LangGraph multi-agent + RAG = real, messy cost drivers, and it ties the two repos together in the story). Alternatives: a toy LangGraph agent, or Despachito if it gains an LLM feature. → *Recommend CloudOracle's Insights Agent.*
+   - **✅ Resolved (done).** Dogfooded on [CloudOracle's Insights Agent](https://github.com/Cro22/CloudOracle) via a LangChain callback shim (ADR D1's proxy fallback — the agent speaks its model natively), CloudOracle source untouched. Gate PASS; the `find-savings` scenario showed a real call-count tail (p95 2.3× median). Harness + findings: [`examples/cloudoracle/`](examples/cloudoracle/).
 2. **Pricing data:** ship a dated snapshot only (v1), or attempt a live fetch?
 3. **CI token-cost tolerance:** how much real spend per CI run is acceptable? This decides whether record/replay is v1 or stretch.
 4. **Proxy vs harness for the runner:** does Jesús want the tool to *drive* the agent (needs an entrypoint contract), or just provide the proxy + headers and let the user run their own harness? (Lighter v1 = the latter.)

examples/cloudoracle/README.md

@@ -0,0 +1,141 @@
+# Dogfooding Augur on CloudOracle's Insights Agent
+
+> Augur measuring a *real* agent, not a synthetic trace: the Insights Agent from
+> [**CloudOracle**](https://github.com/Cro22/CloudOracle) — a LangGraph supervisor
+> multi-agent with tool calls and guardrails.
+
+CloudOracle is Augur's sibling project: FinOps for *cloud* (runtime, post-hoc),
+where Augur is FinOps for *AI agents* (pre-prod, predictive). Dogfooding Augur on
+CloudOracle's own agent closes the loop between the two — and is the first test of
+Augur against an agent it didn't author.
+
+This is the first dogfood: point Augur at an agent it didn't author and see
+whether the cost picture holds up. It does — and it surfaced real findings,
+which is the point of dogfooding.
+
+## The integration: a callback shim, not the proxy
+
+Augur's recording proxy is OpenAI-compatible, but the Insights Agent talks to its
+model **natively** through LangChain (no OpenAI `base_url` to redirect). So we use
+the capture path the [SPEC](../../SPEC.md) documents as the fallback "for
+frameworks where the proxy is awkward" (ADR D1): a **LangChain callback** that
+writes Augur's exact `trace.jsonl` schema from the token usage every call reports.
+
+The agent reuses one chat model across the supervisor, every specialist's ReAct
+loop, and the synthesizer, so a callback on that model sees the **entire call
+graph** of a run — the real agentic fan-out. Coupling is one direction only:
+[`augur_dogfood.py`](augur_dogfood.py) *imports* the agent's public building
+blocks (`build_supervisor_graph`, `build_tools`, `run_guarded`) and builds them
+with our model; **CloudOracle's source is never edited.** Delete this folder and
+both repos are untouched.
+
+## Files
+
+| file | what |
+|---|---|
+| [`augur_dogfood.py`](augur_dogfood.py) | the harness: builds the agent graph with our model + trace callback, runs the scenarios |
+| [`scenarios.yaml`](scenarios.yaml) | representative FinOps questions, spread across the agent's three specialists |
+| [`traffic.yaml`](traffic.yaml) | production volume assumptions for the projection |
+| [`budget.yaml`](budget.yaml) | the thresholds the gate enforces |
+| [`pricing-gemini.yaml`](pricing-gemini.yaml) | a Gemini price snapshot (for the `--provider gemini` path; the Claude run uses Augur's default `pricing.yaml`, which already lists Claude) |
+
+## Running it
+
+Prereqs — bring up CloudOracle's stack so the agent's tools have data to fetch:
+
+```sh
+cd /path/to/CloudOracle
+docker compose up -d postgres                       # pgvector
+go build -o oracle.exe ./cmd/oracle
+# DB defaults (oracle/oracle_dev/cloudoracle) match docker-compose; seed + serve:
+CLOUDORACLE_PROVIDER=synthetic LOG_LEVEL=info ./oracle.exe seed --count 120
+CLOUDORACLE_PROVIDER=synthetic LOG_LEVEL=info ./oracle.exe serve --port 8080 &
+```
+
+Then run the agent under the Augur callback (uses CloudOracle's venv + `.env`):
+
+```sh
+set -a; source insights-agent/.env; set +a       # GEMINI/ANTHROPIC + CLOUDORACLE keys
+unset DATABASE_URL                               # RAG off for a clean first run (see caveats)
+
+insights-agent/.venv/Scripts/python.exe \
+  /path/to/Augur/examples/cloudoracle/augur_dogfood.py \
+  --scenarios /path/to/Augur/examples/cloudoracle/scenarios.yaml \
+  --out /path/to/Augur/cloudoracle-trace.jsonl \
+  --provider anthropic --runs 5 --no-judge
+```
+
+Pipe the real trace through Augur (Claude prices ship in the default snapshot):
+
+```sh
+cd /path/to/Augur
+./augur gate --trace cloudoracle-trace.jsonl \
+    --traffic examples/cloudoracle/traffic.yaml \
+    --budget  examples/cloudoracle/budget.yaml \
+    --pricing pricing.yaml
+
+# And learn the output-length model from the real run:
+cd sidecar && python -m augur_predict fit --trace ../cloudoracle-trace.jsonl \
+    --out ../cloudoracle-model.json --dist quantile
+python -m augur_predict report --model ../cloudoracle-model.json
+```
+
+`--provider gemini` runs the same harness on Gemini (add `--rps 0.15` to stay under
+the free-tier rate limit); `--provider anthropic` (default) uses Claude, whose
+higher limits make a clean multi-call run practical.
+
+## What Augur measured (Claude Haiku 4.5, 20 runs)
+
+```
+scenario "find-savings"  — 5 run(s)
+  metric     mean      p50       p95       stdev     min       max
+  $/run      0.022428  0.016974  0.038833  0.012192  0.016830  0.044236
+  calls/run  6.60      5.00      11.40     3.58      5         13      <-- the tail
+```
+
+| scenario | calls/run (p50→p95) | $/run p95 | note |
+|---|---|---|---|
+| cost-breakdown | 4 → 4 | $0.0069 | input-heavy, cheap output |
+| concept-rightsizing | 4 → 4 | $0.0105 | stable |
+| full-review | 5 → 5 | $0.0197 | the deliberate multi-specialist path |
+| **find-savings** | **5 → 11.4 (max 13)** | **$0.0388** | **the savings specialist sometimes loops** |
+
+**Gate verdict: ✅ PASS** — projected `$/request p95 = $0.0198` (budget $0.02 — it
+*just* fits), `$39.72/tenant/month`, `$794/month` at the assumed volume.
+
+The headline is `find-savings`: its **p95 cost is 2.3× its median**, driven by a
+call-count tail (5 → 13 calls/run) when the savings specialist's ReAct loop keeps
+going. That is precisely the agentic cost driver Augur exists to catch — observed
+on a real agent, not assumed. Gating on the mean would have hidden it; gating on
+p95 (and the wide p95 CI `[$0.019, $0.044]`) surfaces it.
+
+## Findings the dogfood surfaced
+
+Dogfooding earns its keep by what breaks:
+
+- **The agent isn't provider-portable.** Built for Gemini, its LLM-judge layer
+  sends a *system-only* message — which Gemini tolerates but Anthropic rejects
+  (`messages: at least one message is required`). `--no-judge` works around it for
+  the Claude run; the fix belongs in CloudOracle, and Augur found it. (This is a
+  bug in the agent, not in Augur.)
+- **Claude Haiku hallucinates figures on this agent.** Most `find-savings` /
+  `full-review` runs hit the agent's *deterministic grounding* fallback ("answer
+  states $320.00 … not found in any tool result"). A quality signal, orthogonal
+  to cost — Augur still measured the real token spend of those runs.
+- **The sidecar's output model is weak here (R¹ ≈ 0.03).** Output length isn't
+  driven by input length for this agent; it's driven by the call's *role* (a
+  router turn is terse, a synthesis turn is long). The sidecar reports the weak
+  fit honestly rather than faking a trend — and it motivates the documented next
+  step: segment the output model by call role / seq, not just `input_tokens`.
+
+## Caveats / not captured
+
+- **RAG embeddings.** The callback rides the *chat* model; the agent's embeddings
+  go through a separate Gemini embeddings client, so RAG retrieval calls aren't in
+  this trace. `DATABASE_URL` is unset here to keep the run clean. Capturing them
+  would mean instrumenting `GeminiEmbeddingsProvider` too.
+- **Synthetic backend.** CloudOracle serves synthetic resources (`seed`), so the
+  *tool* outputs are representative-shaped, not a real cloud bill. The *agent's*
+  token usage — what Augur measures — is real.
+- **Small N.** 20 runs is enough to see the find-savings tail but thin for a
+  precise p95; the gate's wide CI says so.