feat: add Turn Success Rate metric across evals, realtime, leaderboard

[guohai]

Context

Continues the no-response = NA latency work. Latency correctly excludes no-response runs (a missing response has no latency), but that meant network-impairment failures — where the agent legitimately doesn't respond — could vanish from realtime/leaderboard. Turn Success Rate (TSR) is the quality/resilience axis that captures them: it includes no-response turns as failures, so "responds 0% under impairment" shows up as a low score instead of disappearing.

What TSR measures

A turn succeeds when the agent does the right thing: responds when expected, stops promptly on interrupt, and doesn't false-barge-in. TSR is opportunity-weighted — every such expectation is one scored opportunity, and TSR is the fraction the agent got right:

successful = responses + interrupt-stops + (false-barge chances − actual barge-ins)
total      = response chances + interrupt chances + false-barge chances
TSR = total > 0 ? successful / total : NULL

Each turn counts once (no arbitrary per-dimension priority), and it's computed from the counters computePerCaseAndRates already accumulates. NULL when there are no evaluable turns; behavioral success only (semantic "appropriateness" of the response content isn't judged by the pipeline).

Changes

• Daemon (chunking.ts): turn_success_rate in ComputedRates + the formula; vox-agentd wires it into EvalResult / defaults / rate pickup.
• Schema + migration 0021: nullable turn_success_rate column (registered v22).
• Server: complete route stores it; storage tier select + daily-bucket avg; formatMetricsResults exposes it; api-v1 leaderboard exposes it.
• Leaderboard: TSR column + sort + folded into the composite score. New weights (by end-user experience — noise ranks above naturalness since real-world background noise is constant): Response 25 / TSR 25 / Interrupt 15 / Noise 15 / Network 10 / Naturalness 10. A fully-non-responsive provider now shows NA latency but 0% TSR — visible and correctly penalized rather than hidden.
• Job detail: the "Rates" card is renamed "Turn Success Rate" — composite headline + Response / Interrupt / False Barge-in breakdown.
• Realtime: a Turn Success Rate summary card.
• Tests: opportunity-weighted formula (0.9 fixture, no-response → 0, empty → null); api complete-flow stores + asserts TSR.

Roadmap note (not in this PR)

Network Resilience and Noise Reduction are conceptually the same core metrics measured under adverse conditions — a future Core Experience vs. Resilience restructure (impairment/condition profiles on eval-sets, condition-tagged results, baseline-vs-impaired scoring, run as separate evals) is deferred. This PR keeps the flat 6-metric composite; network/noise remain daemon placeholders today.

Verification

• npm run check clean; 161 unit tests pass (incl. new TSR formula cases).
• Integration: a completion stores + reads back TSR; a no-response run stores TSR = 0 (latency NA).
• Deploy order (same as feat: report "no response" as NA latency, not a zeroed success #67): server migrates first (adds the nullable column, tolerates old daemons that omit it), then upgrade the daemon to emit turn_success_rate.
• The other 9 failing api tests are pre-existing baseline (Cleanup/Clone/Built-in/Version-Gating/mergeEvalConfig) — none in this diff.

Generated with SMT smt@agora.io

[github-actions]

🤖 Claude Code Review

I've reviewed the full diff across schema, migration, daemon, aggregation, API, and UI layers.

Review

Overall this is a clean, well-scoped addition. The new metric is threaded consistently through every layer (schema → migration → daemon parse → both merge paths → raw + bucketed storage selects → MetricSourceRow → realtime/community/my-evals formatter → leaderboard composite → API v1 → three UI surfaces), with tests covering the interesting cases. No security concerns — the change is purely additive numeric data, no new inputs, auth, or credential surface.

Things I verified and found correct:

• Migration registration — 0021_add_turn_success_rate.sql is registered as version 22 in server/migrate.ts, matching the established file-vs-version offset. Nullable real column, no default/backfill, so existing rows read as "no data" — consistent with the null-safe handling everywhere downstream.
• Composite weights still sum to 1.0 (0.25+0.25+0.15+0.15+0.10+0.10). Null handling in the min-max block is correct: TSR gets skipped when no entry has it, and counts weight-but-zero-contribution when one entry is NA against others that aren't.
• TSR math in chunking.ts — numerator terms are all per-case capped (Math.min(len, sampleCount)), so falseSamples - falseReactions ≥ 0 and the ratio can't exceed 1 (the Math.min(1, …) is belt-and-suspenders). totalTurns > 0 ? … : null correctly yields null on empty input. The no-response case (0/5 = 0, not null) is the intended resilience signal and is tested.
• SQL avg(turn_success_rate)::real correctly ignores nulls in bucketed aggregation.
• null vs 0 discipline is respected end-to-end — ?? null in the formatter, naPct/fmtPct render NA for null rather than 0%, matching the latency "NA not 0ms" convention.

Minor observations (not blockers):

• avgRate in routes-api-v1.ts rounds to 4 decimals via Math.round(... * 10000) / 10000 while the main leaderboard uses avgRound(group.turnSuccessRates, 4). Two rounding paths for the same metric — harmless, but worth a shared helper eventually.
• Because the leaderboard composite uses per-metric min-max normalization, TSR at 25% weight amplifies small absolute differences (e.g. 0.97 vs 0.99 spans the full 0→1 axis). This is the pre-existing behavior for every axis, so it's consistent, not a regression — just be aware the 25% weight makes reliability rankings sensitive when providers cluster tightly.

No logic errors, edge-case gaps, or performance regressions found. Good to merge.

[github-actions]

🤖 Codex Code Review

Findings

• Medium: server/routes.ts:3676 / server/routes.ts:3704 and server/routes-api-v1.ts:706 / server/routes-api-v1.ts:723 aggregate TSR by averaging per-result rates. That breaks the PR’s “opportunity-weighted / each turn counts once” definition whenever grouped evals have different opportunity counts; a tiny 1-turn eval and a large 100-turn eval get equal weight. This also gets compounded by daily buckets in server/storage.ts:1165. Consider storing/aggregating TSR numerator + denominator, or deriving them from raw per-case counters before computing the grouped rate.
• Medium: client/src/pages/leaderboard.tsx:88 / client/src/pages/leaderboard.tsx:90 sorts higher-is-better metrics in the wrong direction on the default click path. For turnSuccessRate, comparison = bv - av, but handleSort sets desc, then the code returns -comparison, putting lower TSR first. The new TSR column’s sort will therefore show the worst success rates at the top when users first click it.
• Low: client/src/pages/leaderboard.tsx:140 still says the composite uses 5 metrics and shows the old weights at client/src/pages/leaderboard.tsx:143-client/src/pages/leaderboard.tsx:147. Since this PR adds TSR and changes the weights, the public ranking explanation is now misleading.

No security issues found in the reviewed diff.

[github-actions]

🤖 Codex Code Review

Findings

• server/routes.ts:3704 / server/routes.ts:3756: Missing TSR is treated as worst-score once any entry has TSR. Because the migration does not backfill historical rows and the deploy plan explicitly allows old daemons to omit turnSuccessRate, mixed old/new data will penalize otherwise valid old-daemon results by 25% in the composite. This makes “unknown/unavailable TSR” indistinguishable from “0% success” in the leaderboard. Consider excluding TSR from composite for rows where it is unavailable due to old data, backfilling it, or gating the weight until rollout is complete.
• client/src/pages/leaderboard.tsx:140: The leaderboard info popover still says “5 metrics” with the old 30/25/20/15/10 weights. The actual server composite now uses 6 metrics with TSR 25 / response 25 / interrupt 15 / noise 15 / network 10 / naturalness 10, so users will see incorrect ranking documentation.
• client/src/pages/realtime.tsx:442: providerPrefixFromDataKey only strips _response and _interrupt, so TSR chart keys like provider_tsr_s0 produce a prefix of provider_tsr. The workflow tooltip then looks for provider_tsr_wfname / provider_tsr_wfid, but buildCombinedData stores provider_wfname / provider_wfid. Result: workflow names/links won’t appear on TSR tooltips for Community/My Evals.

I didn’t run tests; review only.

[github-actions]

🤖 Claude Code Review

Reviewed the full diff on the merge ref (23efc34..07f5a7f), including the binary-diffed chunking.ts (which holds the actual TSR computation).

Review

This is a clean, well-tested feature addition. The new turnSuccessRate column is threaded consistently through the schema, migration, storage layer, both metrics endpoints, both leaderboards, and the eval agent — with matching test coverage. No security concerns (no new inputs, auth surfaces, or credential handling).

Correctness — verified sound:

• The TSR math in chunking.ts (successfulTurns / totalTurns) is internally consistent: each numerator term is capped at its denominator (totalResponses ≤ totalSamples, intReactions ≤ intSamples, falseSamples - falseReactions ≥ 0), so the ratio is always in [0,1] and the Math.min(1, …) is purely defensive. The opportunity-weighted model (each interrupt sample contributes both a response and an interrupt opportunity) is coherent and matches the 45/50 = 0.9 test.
• Null handling is correct end-to-end: SQL avg() skips nulls, avgRound/avgRate return null on empty arrays, and the frontend renders NA rather than 0%. The distinction between "no data → null" and "no-response run → 0" is preserved and explicitly tested.
• Leaderboard weights still sum to 1.0 (0.25+0.25+0.15+0.15+0.10+0.10). The generic min-max normalizer already handles the new key via entry[key] with no special-casing needed.
• Migration is registered correctly (version 22 → 0021, matching the existing N→N-1 convention), column is nullable with no default — safe on existing data.

Minor points (non-blocking):

1. Transient leaderboard skew during rollout (server/routes.ts:3758-3763): existing eval_results rows have turn_success_rate = NULL (no backfill). Once some providers have re-run evals with TSR but others haven't, the lagging providers get scored worst (0 contribution, weight still counted) on a 25%-weight axis — so ranking temporarily reflects which provider ran a new eval first, not quality. This is intentionally symmetric with how latency NA is treated, so it's defensible, but the 25% weight makes the effect larger than the latency case. Worth being aware of; a note to stakeholders or a short warm-up window would smooth it.

2. WorkflowTooltip unit prop is untyped (realtime.tsx:452) — the component is already any-typed with an eslint-disable, so this matches existing style; just flagging that the new unit="%" path relies on that.

Nothing here blocks merge.

[github-actions]

🤖 Codex Code Review

Findings

• server/routes.ts:3676 / server/routes.ts:3704 and server/storage.ts:1165: TSR is aggregated by averaging per-result rates, not by pooling opportunities. That violates the stated “every turn counts once” definition when eval jobs have different numbers of response/interrupt/false-barge opportunities; a tiny eval can move the leaderboard as much as a large eval. Consider storing/emitting TSR numerator + denominator, then aggregating sum(successful) / sum(total) for daily buckets and leaderboard rows.

• client/src/pages/leaderboard.tsx:88: Sorting turnSuccessRate uses the existing higher-is-better comparator bug. First-click default is desc for TSR, but comparison = bv - av followed by return -comparison puts lower TSR values first. This makes the new TSR column sort backwards by default. Use one normalized comparator and apply direction once, or explicitly make desc sort bv - av for higher-is-better fields.

Notes

• No security issues found in the diff.
• The core TSR formula in computePerCaseAndRates looks consistent with the PR description for single eval-result computation.

[github-actions]

🤖 Codex Code Review

Findings

• server/routes.ts:3676 + server/routes.ts:3756: Historical/old-daemon rows with turnSuccessRate = null are folded into the new composite as “worst on this axis” whenever any provider has TSR data. Because migration 0021 does not backfill and the deploy plan explicitly allows old daemons to omit TSR, the default 7-day/all-time leaderboard can penalize providers for missing TSR rather than poor behavior. Consider skipping TSR in composite for a window until enough rows have the column, backfilling from rawData.rates where available, or distinguishing “unknown/old schema” from “evaluable but failed”.
• client/src/pages/leaderboard.tsx:88: TSR sorting is reversed for the default higher-is-better direction. comparison is already bv - av for higher-is-better fields, then desc returns -comparison, which sorts low TSR first. This now affects the new “Turn Success” column sort; use a plain numeric comparator and flip only once based on sortDirection.

Notes

• I did not find security issues, injection/auth bypass problems, or obvious performance regressions in the reviewed diff.
• The TSR formula in computePerCaseAndRates matches the PR description and the added tests cover the main no-response/null cases.

[github-actions]

🤖 Claude Code Review

I've reviewed the full diff (including chunking.ts, which renders as binary — see note below). Here's my assessment.

Summary

Clean, well-scoped, and consistently plumbed change. The new turnSuccessRate metric is threaded correctly through schema → migration → daemon → storage aggregation → both metrics routes → all three UI surfaces, with matching null-handling at every layer. The core TSR computation is sound and well-tested (all-success → 1, no-response → 0, empty → null, opportunity-weighting verified numerically). No security issues (no new inputs, injection surface, or auth changes).

Findings

1. chunking.ts — the substantive logic is invisible in the PR diff (reviewability, pre-existing)
The file uses a literal \x00 (NUL) byte as the composite map-key delimiter (${caseId}\x00${chunkId} at line 129), which makes git classify it as binary (Bin 21640 -> 22980). So the actual TSR formula — the heart of this PR — shows only as "Binary files differ" and cannot be reviewed in the diff. The NUL predates this PR (base was already binary), so it's not introduced here, but given that this PR adds real logic to the file it's worth switching the delimiter to an ordinary non-colliding separator (e.g. \u001f is still weird, but even \n or a [caseId, chunkId] tuple key) so future diffs are reviewable. I verified the added logic out-of-band and it's correct.

2. Leaderboard composite: transition-period penalty on a 25%-weighted axis (by design, but heavy)
In server/routes.ts, an entry that is null on TSR while any other entry has a value gets 0 contribution but still counts the full weight (the existing NA-handling). TSR is now weighted 25%, and every result created before migration 0021 has turnSuccessRate = null. During rollout, a provider whose recent mainline results predate the deploy will be scored worst-on-axis for a quarter of its composite until fresh data accumulates, which can visibly reorder the leaderboard. This mirrors the existing latency-NA behavior and is intentional per the code comment, but 25% is a large lever — worth confirming that's the desired rollout behavior (or considering skipping the axis until coverage is broad).

3. Realtime headline TSR reflects only the single latest data point (minor UX)
realtime.tsx:522 uses latest = filteredMetrics[0], so the TSR stat card shows fmtPct(latest?.turnSuccessRate). If that single most-recent result happens to be NA, the card reads "NA" even when recent history has values — unlike the chart beside it, which shows the trend. The other stat cards follow the same latest-based pattern, so this is consistent, just flagging that the headline is a point sample, not an average.

Nothing here is blocking. The metric definition is coherent, the null-vs-zero distinction is handled carefully throughout ("no data" never reads as "0% success"), weights still sum to 1.0, and the migration is properly registered in server/migrate.ts.

[github-actions]

🤖 Claude Code Review

The changes look solid and well-engineered. I reviewed the full diff (schema/migration, daemon TSR computation in chunking.ts, the daemon wiring, both leaderboard routes, and all three frontend pages) plus the tests. No security, correctness, or performance issues.

Review

Correctness — verified sound

• The TSR formula in computePerCaseAndRates (chunking.ts:449-451) is opportunity-weighted and internally consistent: each sample is one response opportunity, and interrupt-phase samples add interrupt/false-barge opportunities on top. I traced the test fixture by hand — successfulTurns = 28 + 9 + (10−2) = 45, totalTurns = 30 + 10 + 10 = 50 → 0.9, matching eval-chunking.test.ts:587. Per-case numerator capping (Math.min(respVals.length, sampleCount)) prevents rates >1, and the totalTurns > 0 guard returns null (not 0) for empty input.
• Null handling is careful and correct everywhere it matters:
  • Realtime/leaderboard aggregation pushes TSR only when != null, so no-data runs don't drag the average to 0 (routes.ts:3673, routes-api-v1.ts:704).
  • Composite score (routes.ts:3752-3772) skips an axis no entry has, but still spends the weight when this entry is NA on a metric others have — the right way to penalize a non-responsive agent without corrupting the min/max range.
  • Leaderboard sort sinks nulls to the bottom in both directions (leaderboard.tsx:84-86), and turnSuccessRate is correctly treated as higher-is-better.
• Migration 0021 is registered as version 22 in server/migrate.ts (consistent with the file-number/version offset), column is nullable with no default, so existing rows backfill as null and are filtered out downstream.

Behavior change worth calling out (not a bug)

• The leaderboard composite weights were re-distributed (response 30→25, interrupt 25→15, noise 20→15, network 15→10, +TSR 25). New weights still sum to 1.0. This silently re-ranks all providers, including historical/cached leaderboard views — intended per the PR, but reviewers/consumers should know the ranking basis moved.

Minor nits (optional)

• realtime.tsx:642 — the new "Turn Success Rate" summary card uses a Clock icon for its info popover trigger. Clock reads as a latency/time affordance; a check/shield/gauge icon would match the metric's meaning better. Cosmetic only.
• TSR pools three dimensions with opportunity weighting, so two providers running different eval-set compositions aren't strictly comparable on TSR — inherent to the metric and shared by the existing responseRate/interruptRate, so acceptable, but a one-line caveat in the leaderboard tooltip wouldn't hurt.

Everything else (the unit prop threading through WorkflowTooltip, the "Custom" provider filter, the visible/total provider-count fix, connectNulls on the TSR chart with undefined gaps) is clean and consistent with surrounding patterns. Tests cover the meaningful cases (all-answered → 1, no-response → 0, empty → null, mixed → 0.9). Good to merge.