Skip to content

v1.8.0

Choose a tag to compare

View all tags
@g8rswimmer g8rswimmer released this 17 Jun 15:53
· 11 commits to main since this release
v1.8.0
6a33423
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What this does

Adds the ability to diff two completed eval runs from the same suite, surfacing regressions and improvements test-case by test-case. This is the EF-04 future consideration from REQUIREMENTS_EVAL.md.

No new database tables are needed — the comparison is computed in-memory from existing TestCaseResult rows using two calls to ListTestCaseResults.

Backend

A new endpoint is registered between the existing GetRun and GetTestCaseResult routes:

GET /v1/eval-runs/{eval_run_id}/compare?baseline_run_id={id}

Both runs must be completed and belong to the same eval suite. The handler fetches TestCaseResult rows for each run, builds lookup maps keyed by test_case_id, then iterates the union to classify each case as one of: regressed (passed → failed), improved (failed → passed), unchanged, new_case (head only), or missing (baseline only). Results are sorted with regressions first, then improvements, then new cases, missing cases, and unchanged — with TestCaseName and TestCaseID as tiebreakers for fully deterministic ordering.

The response includes aggregate counts (regressed_count, improved_count, etc.) alongside the per-case breakdown.

A new migration (0015) adds a UNIQUE KEY (eval_run_id, test_case_id) constraint to eval_test_case_results. This prevents a runner bug from inserting duplicate result rows that would cause non-deterministic comparison output.

Frontend

EvalRunDetailPage gains three new behaviours when the head run is completed:

A baseline selector dropdown appears above the summary panel, listing all other completed runs from the same suite. Selecting one updates the URL as ?baseline_run_id=, making the comparison deep-linkable and bookmarkable. Clearing the selection removes the parameter.

A delta stats banner renders between the summary panel and the results table showing the aggregate counts (-2 regressed, +1 improved, 3 unchanged, etc.).

Per-row change badges appear in EvalRunResultsTable alongside the existing pass/fail indicator — color-coded red for regressed, green for improved, gray for unchanged, indigo for new, and amber for missing.

WorkflowListPage gains an Evals button on each workflow row, linking directly to that workflow's eval suite list.

Code review fixes applied

After an initial implementation review, the following issues were addressed in a follow-up commit:

Stale fetch cancellation. The compare fetch effect now carries an alive flag (matching the pattern used by the existing poll effect). When the user clears the baseline selector before the in-flight fetch resolves, the response is silently dropped rather than overwriting the cleared state.

Failed run guard. The frontend previously fired compareEvalRuns for both completed and failed head runs. The backend rejects anything that is not completed, so this always produced a confusing error banner. The guard is now run?.status === 'completed' only.

Sibling loader pagination. listEvalRuns defaulted to 50 results. A suite with more than 50 completed runs would silently omit older baselines from the dropdown, even when a deep-linked ?baseline_run_id pointed to one of them. The sibling loader now passes limit=200.

Sibling loader freshness. The sibling loader effect previously depended only on run?.suite_id and runId, both of which are stable after initial load. Adding run?.status to the dependency array causes the loader to re-fetch when the head run transitions to completed, so a run that finishes in another tab appears in the selector without a page reload.

Sort stability. sort.Slice is not guaranteed stable. The comparator was upgraded to sort.SliceStable with TestCaseID added as a tertiary key, ensuring identical requests always return the same ordering. The changeOrder lookup map was also hoisted to package scope to avoid a heap allocation on every request.

Files changed

Backend

  • backend/internal/eval/handler.go — CompareRuns handler, types, and constants
  • backend/internal/api/router.go — route registration
  • backend/internal/eval/compare_test.go (new) — 13 table-driven tests covering all change classifications and all validation error paths
  • backend/internal/store/mysql/migrations/0015_unique_eval_test_case_result.up.sql (new)
  • backend/internal/store/mysql/migrations/0015_unique_eval_test_case_result.down.sql (new)

Frontend

  • frontend/src/api/types.ts — CompareChangeType, TestCaseComparison, EvalRunCompare
  • frontend/src/hooks/useApi.ts — compareEvalRuns, updated listEvalRuns with optional limit
  • frontend/src/pages/EvalRunDetailPage.tsx — baseline selector, compare effect, delta banner
  • frontend/src/components/eval/EvalRunResultsTable.tsx — optional compareMap prop and change badges
  • frontend/src/pages/WorkflowListPage.tsx — Evals button on each workflow row

Docs

  • DEMO_EF04.md (new) — end-to-end walkthrough of all comparison scenarios using curl
Assets 2
Loading