Design: outcome / quality signal for 'same output, less spend' comparisons

[willwashburn]

This is a design issue, not an implementation ticket.

Before building, we need a decision.

Problem

Burn's meta-goal is answering "would the same work have cost less with a different model / harness / tool choice?" Spend data alone can't answer this — we also need a notion of whether the cheaper alternative produced an equivalent result. Otherwise "Haiku would have cost 5× less" is useless if Haiku would have produced broken code.

This is the moat versus every other usage tracker (TokenTracker, ccusage, Cursor stats). None of them have a quality axis. Whichever signal we pick, the data model must accommodate it from the start so it doesn't become a v2 retrofit.

Candidate signals

(a) Explicit outcome tag at session end

burn tag --outcome good|bad|meh [--note \"…\"]
burn claude --tag workflow=refactor -- <claude args>
  # on exit, prompts: \"how did this go? [good/bad/meh/skip]\"

• Pros: simplest, most reliable when present, user controls the semantics.
• Cons: relies on discipline. Realistic capture rate probably 20–40% without prompts. Even with prompts, users will skip during flow.

(b) Git-state scrape

Record git HEAD, working-tree status, and test-suite state (npm test, pytest, etc. — whatever's there) at session start and end. Later: check whether the session's commits were reverted within N days, whether a PR containing them merged, whether tests still pass on HEAD.

• Pros: automatic. Captures the real-world "did this stick" signal.
• Cons: test harness detection is language/project-specific. Revert-within-N-days requires a delayed check (cron or on-demand). Won't capture quality for non-code tasks.

(c) Benchmark harness

Out of scope for v0.1 but worth considering in the data model. Re-run the same prompt across models with automated scoring. Requires storing prompts — which we explicitly don't do today for privacy.

(d) Do nothing

Report cost and let the user compare quality by hand. Simplest, ships fastest, but forfeits the differentiator.

Hard constraint

Burn does not store prompts or responses — privacy is a deliberate design choice. Whatever signal we pick must not require storing them. This rules out (c) in its naive form; any benchmarking would need explicit opt-in with its own storage story.

Asks of this issue

1. Pick one (or a hybrid — (a)+(b) is the obvious pairing).
2. Shape the data model now so TurnRecord / the ledger can carry an outcome without a schema migration later. Candidate: a sidecar outcomes.jsonl indexed by (sessionId, workflowId?, projectKey) with { signal, value, capturedAt, confidence? }.
3. File implementation sub-issue(s).

Decision criteria

• How many real sessions will actually produce an outcome signal? (If (a) alone captures < 20%, it's probably not worth the UX cost.)
• Does the signal survive being wrong? (Test suites lie; user tags are subjective; revert rate has false positives.)
• How much does it cost to ignore this for now and bolt on later?

Meta: the cheapest path may be (a) as the MVP with (b) as a later automatic augment — but worth arguing through before committing.

[willwashburn]

Candidate (e) considered and rejected as primary: automatic adherence detection à la prism.

Prism detects when CLAUDE.md rules stop being followed mid-session without user tagging. Mechanism (analyzer.py:507-583): extract lines starting with Never|Always|Don't|Use|Avoid|Run|Prefer|Must|Ensure, then run hand-coded per-rule checkers (e.g. look for migration file edits when rule says don't touch them; look for interactive Bash; look for TypeScript any patterns).

Why not primary:

• Each rule needs its own checker function. Doesn't scale to arbitrary CLAUDE.md content.
• Detector silently returns 0 violations when a rule can't be auto-checked (analyzer.py:583) — high false-negative rate, no signal of coverage.
• Narrow domain (JS/TS/migrations patterns in prism's case). Useful for prism's target audience but not generalizable to burn's integrators (relay, workforce) across arbitrary workflows.

Where it might still be useful: as a supplementary signal alongside (a) or (b). If we ever ship CLAUDE.md hot-path analysis (separate issue), the same section-parsing could surface "you have a rule at line 47 that prism-style checkers could validate if written." User opts into authoring the checker. Defer until demand is observed.

Current primary-candidate ranking (unchanged): (a) explicit tag at session end, (b) git-state scrape, hybrid of (a)+(b) as MVP. (c) benchmark harness and (e) automatic adherence remain future options.

[willwashburn]

Strong candidate surfaced from agentsview — outcome inference from session metadata + last-assistant-text patterns.

Source: internal/signals/outcome.go:1-95 in agentsview. 95 lines, zero deps beyond strings + time. Pure computation over:

• IsAutomated
• MessageCount
• EndedWithRole — \"user\" or \"assistant\"
• FinalFailureStreak
• LastAssistantText — for give-up pattern matching
• LastActivity — for recency check

Classification output: completed | abandoned | errored | unknown, each with confidence high | medium | low.

Why this beats the earlier candidates

• Better than explicit tagging (a): automatic — no user discipline required. Works retroactively on every past session.
• Better than git-state scrape (b): simpler, language-agnostic, doesn't require test harness detection.
• Better than adherence detection (rejected earlier): general heuristics, not hand-coded per-rule checkers. Scales.

Classification rules (cribbed directly)

if IsAutomated:                           → unknown (low)
if MessageCount == 2 && ended=assistant:  → completed (medium)
if MessageCount < 3:                      → unknown (low)
if isRecent(LastActivity, 10min):         → unknown (low, recent=true)
if ended=user:                            → abandoned (medium, or high if ≥10 msgs)
if FinalFailureStreak ≥ 3:                → errored (medium)
if ended=assistant:                       → completed (medium, downgrade to low if give-up pattern)

Give-up patterns observed: \"i'm unable to\", \"i can't proceed\", \"i don't have access\", \"i cannot proceed\", \"i am unable to\".

Content dependency

Requires LastAssistantText for give-up-pattern matching. With #33 (content sidecar) the text is available directly. Without it, the classifier still works — just degrades the "give-up downgrade" from medium-to-low confidence, treating all assistant-ended sessions as completed at medium confidence.

Proposed resolution of this design issue

Adopt outcome inference as the primary quality signal. Keep (a) explicit tagging as an optional enhancement for users who want to override the classifier. Drop adherence-detection entirely (already rejected in a comment).

Data model: one SessionOutcome record per session, computed lazily at query time (not stored). Stored at query time on demand, not ingested.

interface SessionOutcome {
  sessionId: string;
  outcome: 'completed' | 'abandoned' | 'errored' | 'unknown';
  confidence: 'high' | 'medium' | 'low';
  isRecent: boolean;
}

Sub-issue to follow

Once this design is confirmed, file the implementation as packages/analyze/src/outcome.ts with the classifier + burn summary --with-outcomes flag.

Closing this design issue after the decision is made — not before.

[willwashburn]

Second primary candidate surfaced from codeburn (#37 activity classifier).

One-shot rate as a quality signal.

Definition: oneShotRate = oneShotTurns / editTurns where an edit turn is "one-shot" if it has edits AND zero retries (no edit→bash→edit cycles).

Source: codeburn's classifier.ts:120-140 (countRetries) and compare-stats.ts:49.

Advantages over outcome inference (agentsview's approach)

Not competing — complementary. Each captures a different failure mode:

Signal	Captures	Needs content?	Granularity
Outcome inference (agentsview)	abandoned sessions, give-up sessions	Yes (last assistant text)	Per-session
One-shot rate (codeburn)	retry-heavy sessions, stuck-in-loop sessions	No — tool patterns only	Per-turn

Running both gives a more complete picture. A session can be:

• High one-shot rate + completed outcome → clean success
• Low one-shot rate + completed outcome → got there but inefficiently (model cost much higher than needed)
• Any one-shot rate + abandoned outcome → user gave up

The second case is particularly interesting for burn's meta-goal: it's exactly the "Sonnet nailed it first try" vs "Haiku got there after 3 retries" distinction that drives model-choice decisions.

Content dependency

One-shot rate requires no content. Works entirely from TurnRecord.toolCalls + the retry counter on the turn. Robust in content.store=hash-only mode where outcome inference degrades.

Proposed resolution

Adopt both as primary signals:

1. Outcome inference — per-session signal (completed/abandoned/errored with confidence)
2. One-shot rate — per-turn signal (aggregated per session, per workflow, per model, per activity category)

Sub-issue for implementation: packages/analyze/src/quality.ts with both classifiers. burn summary --quality surfaces both.

One-shot rate falls out naturally when #37 (activity classifier) ships — the classifier already computes retries and hasEdits per turn; one-shot rate is a trivial query over those fields.

Updated candidate ranking

• (a) Explicit tagging — demoted to optional enhancement
• (b) Git-state scrape — still a future option
• (c) Benchmark harness — still out of scope
• (d) Do nothing — rejected
• (e) Automatic adherence (prism) — rejected (see earlier comment)
• (f) Outcome inference (agentsview) — primary (from earlier comment)
• (g) One-shot rate (codeburn) — primary (this comment)

(f) + (g) is the MVP. Close this design issue with that decision once (f) and (g) both land as concrete implementations.