tml-2720: two-tier scorecard + token/correctness trace vocabulary#640
Merged
Conversation
Scaffold the "Drive — Judge + live-experiment harness" project workspace: two-tier correctness-first scorecard, an LLM judge calibrated against an accreting instrumented-run corpus, and an SDK-spawned k=N A/B harness. Four slices (TML-2720 scorecard+vocabulary, TML-2735 golden-case harness, TML-2736 judge, TML-2737 experiment engine); two foundation slices run in parallel, judge + engine stack on top. Trace.jsonl carries the first natively-instrumented project-started/spec-authored/plan-authored events, emitted via the deterministic emitter merged in PR #633. Signed-off-by: wmadden-electric <286902546+wmadden-electric@users.noreply.github.com>
… spike frameworks Operator steer: keep the implementation minimal. Default to a bespoke LLM-judge + held-out agreement tally; adopt a third-party eval framework (Inspect/Braintrust/promptfoo) only if a time-boxed slice-3 spike shows it reduces net complexity. Run-production harness stays bespoke regardless. Adds spec non-goal + Open Question 6, design-notes alternative, plan slice-3 spike note, and the spike to TML-2736. Trace carries spec-amended/plan-amended. Signed-off-by: wmadden-electric <286902546+wmadden-electric@users.noreply.github.com>
Settle the six project-level open questions into decisions: - one project (judge + harness kept together; the feed->consume loop is the project) - judge model cross-family (hard); default GPT 5.5 vs the Claude orchestrator - per-run token signal from the SDK TurnEndedUpdate.usage - composed correctness gate (validation gates + QA run + judge intent); CI/merge is real-PR-only since sandboxed runs cannot use CI without an isolated fork - QA plans pre-written in each golden case acceptance set - baseline = previous skill version - bespoke-minimal scorer; slice-3 spike gates any framework on a net-complexity win Open Questions section now empty; decisions logged in spec + design-notes. Trace carries spec-amended. Signed-off-by: wmadden-electric <286902546+wmadden-electric@users.noreply.github.com>
Signed-off-by: Will Madden <madden@prisma.io>
… events Signed-off-by: Will Madden <madden@prisma.io>
…le verdict Signed-off-by: Will Madden <madden@prisma.io>
…ster test Signed-off-by: Will Madden <madden@prisma.io>
Contributor
|
Warning Review limit reached
More reviews will be available in 23 minutes and 37 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (7)
📒 Files selected for processing (11)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
size-limit report 📦
|
@prisma-next/extension-author-tools
@prisma-next/mongo-runtime
@prisma-next/family-mongo
@prisma-next/sql-runtime
@prisma-next/family-sql
@prisma-next/extension-arktype-json
@prisma-next/extension-cipherstash
@prisma-next/middleware-cache
@prisma-next/mongo
@prisma-next/extension-paradedb
@prisma-next/extension-pgvector
@prisma-next/extension-postgis
@prisma-next/postgres
@prisma-next/sql-orm-client
@prisma-next/sqlite
@prisma-next/target-mongo
@prisma-next/adapter-mongo
@prisma-next/driver-mongo
@prisma-next/contract
@prisma-next/utils
@prisma-next/config
@prisma-next/errors
@prisma-next/framework-components
@prisma-next/operations
@prisma-next/ts-render
@prisma-next/contract-authoring
@prisma-next/ids
@prisma-next/psl-parser
@prisma-next/psl-printer
@prisma-next/cli
@prisma-next/cli-telemetry
@prisma-next/emitter
@prisma-next/migration-tools
prisma-next
@prisma-next/vite-plugin-contract-emit
@prisma-next/mongo-codec
@prisma-next/mongo-contract
@prisma-next/mongo-value
@prisma-next/mongo-contract-psl
@prisma-next/mongo-contract-ts
@prisma-next/mongo-emitter
@prisma-next/mongo-schema-ir
@prisma-next/mongo-query-ast
@prisma-next/mongo-orm
@prisma-next/mongo-query-builder
@prisma-next/mongo-lowering
@prisma-next/mongo-wire
@prisma-next/sql-contract
@prisma-next/sql-errors
@prisma-next/sql-operations
@prisma-next/sql-schema-ir
@prisma-next/sql-contract-psl
@prisma-next/sql-contract-ts
@prisma-next/sql-contract-emitter
@prisma-next/sql-lane-query-builder
@prisma-next/sql-relational-core
@prisma-next/sql-builder
@prisma-next/target-postgres
@prisma-next/target-sqlite
@prisma-next/adapter-postgres
@prisma-next/adapter-sqlite
@prisma-next/driver-postgres
@prisma-next/driver-sqlite
commit: |
4 tasks
wmadden
added a commit
that referenced
this pull request
May 31, 2026
…#654) ## Linked issue Refs [TML-2736](https://linear.app/prisma-company/issue/TML-2736). Third slice of the **Drive — Judge + live-experiment harness** project; builds on the two-tier scorecard ([#640](#640)) and the golden-case harness ([#641](#641)). ## At a glance The judge grades one Drive run and emits the `intent` correctness signal the scorecard already reads. Two invariants do the load-bearing work. A malformed model response is never a silent pass: ```ts const validated = RubricResponse(parsed); if (validated instanceof type.errors) { return { intent: null, reasons: [`malformed model output: ${validated.summary}`] }; } ``` …and the emission preserves any gate-recorded `mechanical`/`qa` rather than clobbering it, because the scorecard is last-write-wins on the whole triple: ```ts export function mergedCorrectnessPayload(events, projectRunId, intent) { const prior = latestCorrectness(events, projectRunId); return { mechanical: prior?.mechanical ?? null, qa: prior?.qa ?? null, intent }; } ``` Before this slice, the scorecard's `intent` component was always `null` → every run was `not-computable`. This slice makes it producible. ## Summary This PR ships a **bespoke-minimal LLM judge** under `skills-contrib/drive-judge-harness/judge/`. It carries two substantive pieces: 1. **The judge itself** — grades a completed Drive run (the produced diff + the run's trace) against a golden case's `acceptance.md`, through a cross-family judge model, and emits the `intent` correctness component. Three prompt sets: a requirements+intent rubric, a failure-mode classifier, and an operator-turn classifier. 2. **The recorded decision to build it bespoke** — a time-boxed spike compared Inspect / Braintrust / promptfoo and confirmed bespoke-minimal. The rationale lands in the project `spec.md` and `design-notes.md`; promptfoo is the recorded escape hatch. The judge model is **injected** everywhere, so the whole subtree typechecks, tests, and lints with **no `CURSOR_API_KEY`** and **`@cursor/sdk` absent** — tests pass a mock. The live adapter is reached only behind the same `--live` + key gate as the harness. ## How it fits together 1. **The model boundary** (`judge/judge-model.ts`) — a one-method `JudgeModel` interface (`grade(prompt) => Promise<string>`). Everything downstream takes it as a dependency; tests inject a mock and never make a real call. 2. **The live adapter** (`judge/judge-model-sdk.ts`) — pins a cross-family judge id (default `gpt-5.5`) and **rejects a same-family judge id at construction** (a Claude judge grading a Claude orchestrator throws before any SDK code runs). The `@cursor/sdk` import is lazy, so module load stays green without the package. 3. **The three prompt sets** (`rubric-correctness.ts`, `classify-failure.ts`, `classify-operator.ts`) — each renders a prompt, calls the model, and parses an arktype-validated verdict. The operator-turn classifier uses the measurement model's five canonical buckets (`docs/drive/measurement-model.md`): legitimate-design, legitimate-authorisation, illegitimate-asked, illegitimate-correction, illegitimate-rescue. 4. **The merge-preserving emission** (`judge/emit-correctness.ts`) — folds the rubric's `intent` into the run's latest recorded `{mechanical, qa}` and emits one `correctness-recorded` event through the deterministic emitter. The slice-1 scorecard composes it; no scorecard or schema edits. 5. **The calibration harness** (`judge/calibration.ts` + `judge/calibration/labels.md`) — a judge-vs-human agreement tally with a ≥0.80 gate. The machinery lands; the calibration *run* is parked (see Reviewer notes). ## Reviewer notes - **The calibration run is deliberately parked, not forgotten.** Calibration needs ~10–20 instrumented runs, and corpus generation is real-dollar spend the operator is holding. So this slice ships the gate machinery and an honest "uncalibrated" status; the project-DoD calibration item stays unchecked. `SKILL.md` and `calibration/labels.md` both record the deferral and the operator-spend gate. - **The merge rule is the subtle part.** `computeScorecard` is last-write-wins on the whole `{mechanical, qa, intent}` triple — it does not merge components. A naive judge emitting `{mechanical:null, qa:null, intent:pass}` would erase a gate's recorded pass. `emit-correctness.ts` reads-merges-emits so that can't happen; the end-to-end test asserts a prior `mechanical:pass` survives. - **One unplanned helper.** `judge/parse-json.ts` lifts a JSON object out of a model response (bare / fenced / embedded) — factored out so the malformed-→null path lives in one place rather than three copies. - **The planning commit rides along.** The first commit scaffolds the slice (spec, plan, trace) and records the spike; the second is the implementation. They're one reviewable unit. ## Testing performed - `node --test` over the six new judge suites — **43 cases, all green**, run with `CURSOR_API_KEY` unset. - `pnpm typecheck` — clean. - `pnpm lint:deps` — no dependency violations. - `pnpm lint:casts` — `delta=0` (no new bare casts). - `pnpm test:scripts` — 545 cases green (nothing else regressed). ## Skill update `skills-contrib/drive-judge-harness/SKILL.md` documents the judge, the cross-family requirement, the `correctness-recorded` merge rule, the fail-to-null invariant, and the parked calibration. ## Checklist - [x] DCO sign-off on every commit - [x] Tests written first and passing - [x] Title follows the `TML-NNNN:` convention - [x] No new bare casts (`lint:casts` delta 0) ## Alternatives considered - **Adopt an off-the-shelf eval framework (Inspect / Braintrust / promptfoo).** Confirmed-rejected by the spike. They grade `(input → model output)`; our unit is a whole Drive run scored from trace + diff + golden acceptance set. A framework can host the tiny grading call but not the integration with our trace/scorecard/golden assets — that glue is bespoke either way. promptfoo (TS, MIT, local) is recorded as the escape hatch if the bespoke scorer grows hairy. - **Emit the `intent` component on its own.** Rejected — it would clobber the gate-recorded `mechanical`/`qa` under last-write-wins. Hence the merge-preserving helper. - **An `other` operator-turn bucket + non-null fallback.** Rejected — the measurement model defines exactly five buckets; a malformed response yields `bucket: null` (same fail-to-null discipline as the rubric) rather than an off-doc catch-all. - **Run the calibration now.** Rejected — corpus generation is held on cost. The judge ships uncalibrated-but-honest; the gate is computable the moment the corpus exists. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit # Release Notes * **New Features** * Introduced LLM-based judge system for Drive orchestrator evaluation with failure mode classification, operator turn assessment, and correctness rubric grading * Implemented cross-family model constraint enforcement between judge and orchestrator * Added calibration framework for judge accuracy validation with agreement-rate metrics * **Documentation** * Expanded judge harness documentation with detailed module descriptions and key invariants * Added calibration corpus specification and workflow guidance * **Tests** * Added comprehensive test coverage for judge components, classifiers, and calibration logic <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Will Madden <madden@prisma.io> Co-authored-by: Will Madden <madden@prisma.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The decision: the report must refuse to imply "good" without a correctness signal
drive-diagnose-runprinted twenty disaggregated metrics and a static "Not computable" caveat, but nothing bound correctness to efficiency — and the token row claimed "not instrumented". A reader skimming all-green metrics could quietly conclude the run was good. This PR makes that impossible: the report headline is now a two-tier scorecard, and when no external correctness signal is present the verdict line readsnot computableand names the missing input rather than staying silent.not computableis the correct, shippable state of the scorecard for the entire window before the judge exists — it is not a stub.Two-tier scorecard
project_run_id, the gate reads the externalcorrectness-recordedfeed (mechanical/qa/intent). All threepass→CORRECT; anyfail→INCORRECT; anynullor a missing feed →not computable, naming the missing component(s) (or "external correctness signal" when the feed is absent entirely).tokens-recordedfeed), wall-clock, and rework — rendered only over runs that passed Tier 1, and hidden with a one-line reason otherwise. Null/absent token figures rendern/a (no signal). Scoring an incorrect or ungraded run's efficiency is meaningless, so Tier 2 is gated on Tier 1.New module
scorecard.tscomputes the scorecard;report.tsrenders it as the headline (replacing the old static verdict block) and the stale "token usage: not instrumented" operator row is gone.Trace vocabulary additions
Two per-run event types added to the single canonical schema (
skills-contrib/drive-record-traces/schema.ts), the union, andKNOWN_EVENT_TYPES; both documented inevents.md:tokens-recorded—input_tokens/output_tokens/cache_read_tokens/cache_write_tokens, eachinteger ≥ 0 | null, sourced from the Cursor SDK'sTurnEndedUpdate.usage(snake_case to match the vocabulary; mapped 1:1 to the SDK's camelCase). Hand-runs never emit it.correctness-recorded— the external Tier-1 verdict slot the judge will populate:mechanical/qa/intent, each"pass" | "fail" | null. This PR builds only the slot.Both feeds are emitted after and outside the orchestrated run (the harness accumulates tokens; the judge grades correctness post-hoc), so they are discrete per-run events rather than fields on a lifecycle event whose writer isn't present when the value becomes known.
Scope — deliberately deferred to later slices
correctness-recorded(slicellm-judge, TML-2736).experiment-engine, TML-2737).golden-case-harness, TML-2735) —posthoc.tsuntouched, no@cursor/sdkdependency, no golden cases.Tests + gates
Test-first throughout (
node --test). New:scorecard.test.ts(verdict classification, missing-input naming, token aggregation over CORRECT runs);emit.test.tscases (accept well-formed / reject malformed for both new events);report.test.tscases (not computable+ named missing signal, Tier-2 hidden for non-correct runs,n/a (no signal)for null tokens).Green:
pnpm lint:deps,pnpm lint:casts(delta 0),pnpm test:scripts(467 tests incl. the registeredscorecard.test.ts), and a directtsc --noEmitover the editedskills-contribfiles (these aren't in the turbotypecheckgraph). The scorecard renders the honestnot computableverdict end-to-end over this slice's own trace.Stacking
Stacked on planning PR #638 (base
main). This branch carries #638's planning commits until #638 merges; review the four commits fromDrive(scorecard-and-trace-inputs): slice spec + planonward.Linear: TML-2720.