revert(results): restore eval results to state at 4e50650

[claude]

Requested by Matt Rossman · Slack thread

Before

The public evals site's Benchmark table shows a Claude Code / Opus 4.8 (high) row that only has Build data. Deploy, Investigate, and Resolve render as em-dashes (no data), yet the row still reports a 100% TOTAL. It's backed by just two Build-stage records, so the total is misleading to anyone reading the public table.

After

That row is gone and the Benchmark table matches the exported results as of 4e50650.

How

Restores apps/web/src/data/eval-results.json byte-for-byte to its state at commit 4e50650 (chore: refresh eval results (#174), 2026-08-06 07:18 UTC). Scope is deliberately limited to that one file — regression-suite results live separately in apps/web/src/data/regression-eval-results.json and are not touched by this PR. Diffstat: 1 file changed, 620 insertions(+), 3224 deletions(-).

#129 (9bd7320, 2026-08-06 12:51 UTC) is the only commit to touch eval-results.json since 4e50650, which matches Matt's diagnosis.

Structural diff

Record count goes 220 → 190. All records in both versions are suite: "benchmark". Of the 182 records present in both, zero differ — no pass/fail flips, no field changes. The entire delta is the rows below.

Removed — the incomplete Opus 4.8 row (4 records). claude-code-opus-4.8 and claude-code-opus-4.8-no-skills, each holding only build-cli-001-bootstrap-app and build-database-001-migrate-postgres-to-supabase (both stage: build, both passing) — exactly the shape that produces Build 100% / em-dashes / 100% TOTAL. Supporting evidence that this data was accidental: there is no experiments/claude-code-opus-4.8*.ts definition in the repo, so no refresh run could have generated it. The only other mention of claude-opus-4-8 anywhere is a label formatting assertion in apps/web/src/lib/format.test.ts.

Removed — results for three new supabase-js SDK evals across the other 14 benchmark rows (34 records). ⚠️ This is the part that goes beyond the reported bug and is worth a second opinion:

• build-auth-001-email-password-flow
• build-dataapi-001-relational-report
• build-dataapi-002-restock-alert-report

Affected rows: claude-code-opus-5, claude-code-sonnet-5, codex-gpt-5.4-mini, codex-gpt-5.6, opencode-kimi-k3, and each of their -no-skills counterparts. Their eval definitions stay in evals/ (this PR touches only the JSON), so the next benchmark refresh should repopulate them — they'll just be absent from the site until it runs.

Restored — metadata re-attributions that #129 introduced (8 records). Not new data; the same two Build evals moving back to their pre-#129 identity:

Eval	Currently on main	After this PR
build-cli-001-bootstrap-app, build-database-001-migrate-postgres-to-supabase	claude-code-opus-4.8 / claude-opus-4-8	claude-code-opus-5 / claude-opus-5
same two, -no-skills	claude-code-opus-4.8-no-skills	claude-code-opus-5-no-skills
same two, codex	codex-gpt-5.6 @ effort low	codex-gpt-5.6 @ effort medium
same two, codex -no-skills	codex-gpt-5.6-no-skills @ low	codex-gpt-5.6-no-skills @ medium

In-flight refresh — possible conflict

The benchmark refresh run started around 16:54 UTC today (run 31121347612) had not landed on main as of 85c4f6a. Per .github/workflows/eval-refresh.yml, scheduled/manual runs open their own PR on chore/refresh-eval-results-* rather than pushing to main. If that run produces a refresh PR touching eval-results.json, it will conflict with this branch and whichever merges second wins — so it's worth checking for that PR before merging. A refresh should regenerate the three SDK evals and should not reintroduce the Opus 4.8 row.

Test plan

• Confirm the Benchmark table on a preview build no longer shows a Claude Code / Opus 4.8 row
• Confirm remaining rows' Build/Deploy/Investigate/Resolve/TOTAL figures match pre-#129 values
• Decide whether losing the three SDK evals' results until the next refresh is acceptable, or whether a forward fix (drop just the Opus 4.8 records and repair the re-attributions) is preferred

---

Generated by Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
evals	Ready	Preview	Aug 6, 2026 6:23pm

[claude]

Marking ready for review. One note for whoever merges: mergeable_state is blocked because the Code Quality check is missing, not because of a content problem.

Today's GitHub Actions incident meant no workflow run was ever dispatched for this branch (/actions/runs?branch=revert/eval-results-4e50650 returns total_count: 0; the newest run anywhere in the repo is 17:53 UTC, before this PR was opened at 18:23). A draft→ready transition doesn't fire on: pull_request, and main has no commits this branch lacks, so there was no legitimate push available to re-trigger it. Only the third-party checks (Vercel, [code]smith) reported.

I ran the check locally on 73be43a instead — the exact command from .github/workflows/biome.yml:

$ pnpm format:check          # → biome check . && pnpm --filter @supabase-evals/web format:check
Checked 191 files in 72ms. No fixes applied.
Found 2 warnings.
All matched files use Prettier code style!
# exit 0

Passes. (The 2 warnings are broken symlinks under skills/ from uninitialized submodules — unrelated to this PR and present on main.)

Worth knowing: format:check would not have inspected this PR's file even if it had run. biome.json lists apps/web in files.ignore, and @supabase-evals/web's format:check is prettier --check "**/*.{ts,tsx}" — no JSON glob. Running biome directly on the file confirms it: Checked 0 files. So formatting was never a failure risk here. I separately verified the file parses as valid JSON and is byte-identical to 4e50650 (git diff 4e50650 HEAD -- apps/web/src/data/eval-results.json is empty).

No push and no commit were made — the head is still 73be43a. To get a green Code Quality badge, this needs either an admin override or a re-run once Actions recovers.

---

Generated by Claude Code

[mattrossman]

Confirmed the numbers look the same as what I saw on the older preview https://evals-git-revert-eval-results-4e50650-supabase.vercel.app/ without Opus 4.8 showing anymore. Merging this as a hotfix to fix prod results.