feat: supabase-js SDK evals (auth flow, relational report, key migration) - #129
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
f1b1b25 to
216ca87
Compare
mandarini
added
run-evals-changed
mandarini
added
the
run-evals-changed
mandarini
removed
the
run-evals-changed
mandarini
added
run-evals-changed
barryroodt
left a comment
barryroodt
left a comment
There was a problem hiding this comment.
These four are in good shape, and pulling the expected numbers out of the seeded migrations instead of hardcoding them is the right call. The auth driver holding one session across signup and both signin attempts is nice too.
One thing to sort before merge: implementation uses @supabase/supabase-js is the only failed check in all 13 benchmark failures, and the grep behind it doesn't prove SDK use. The pattern requires a quote before the package name but nothing after it (EVAL.ts:198), so @supabase/supabase-js-not-real matches, and an unused import passes just as well as a real one. So those three evals currently score whether a string appears somewhere under app/, which makes the codex/claude split hard to read.
My call is to fix the detection rather than name the SDK in the prompt, since unprompted reach is the thing you're measuring and that only means something if the gate is real. Resolving the import from the scored entry, or instrumenting module load, would both do it.
Rest is inline. The resolve-sdk-001 one is worth a look, a hardcoded stub currently passes it 4/4.
a3024b3 to
1a06c3d
Compare
mandarini
added
the
run-evals-changed
|
@barryroodt thanks for the review! addressed your comments. Can you please take a look? |
mandarini
removed
the
run-evals-changed
barryroodt
left a comment
barryroodt
left a comment
There was a problem hiding this comment.
Nice, this is a lot tighter. Two things before merge.
The new key-dependency checks on resolve-sdk-001 only ask whether the corrupted run still prints parseable JSON, so a valid migration that handles the API error and prints [] fails with a misleading "hardcoded" result. Comparing against the expected titles and draft count you already compute fixes it.
build-dataapi-002-restock-alert-report has no result rows at this head, down from 8 (2 passing) at a3024b3, so I think the rebase dropped them. The rows that remain predate 488a985, so one refresh after that fix covers both.
Everything else from the first review is addressed. Nits are inline.
|
Your refresh landed while I was writing that, so scratch my second point. It confirms the first point though, both resolve-sdk rows now fail on the two new checks. More on that thread. |
mandarini
added
run-evals-changed
|
@barryroodt pushed one more commit! |
barryroodt
left a comment
barryroodt
left a comment
There was a problem hiding this comment.
Nice, that's all of it. Corrupting the URL instead of the key is a better call than what I suggested.
resolve-sdk-001 is 2/2 on the refreshed rows, both new checks reporting broke as expected against an unreachable project URL.
<!-- ccr-slack-attribution --> _Requested by **Matt Rossman** · [Slack thread](https://supabase.slack.com/archives/C051L8U2EJF/p1786039678828039?thread_ts=1786039678.828039&cid=C051L8U2EJF)_ ## Before The public evals site's Benchmark table shows a **Claude Code / Opus 4.8 (high)** row that only has Build data. Deploy, Investigate, and Resolve render as em-dashes (no data), yet the row still reports a **100% TOTAL**. It's backed by just two Build-stage records, so the total is misleading to anyone reading the public table. ## After That row is gone and the Benchmark table matches the exported results as of `4e50650`. ## How Restores `apps/web/src/data/eval-results.json` byte-for-byte to its state at commit [`4e50650`](4e50650) (`chore: refresh eval results (#174)`, 2026-08-06 07:18 UTC). Scope is deliberately limited to that one file — regression-suite results live separately in `apps/web/src/data/regression-eval-results.json` and are **not** touched by this PR. Diffstat: `1 file changed, 620 insertions(+), 3224 deletions(-)`. `#129` (`9bd7320`, 2026-08-06 12:51 UTC) is the only commit to touch `eval-results.json` since `4e50650`, which matches Matt's diagnosis. ## Structural diff Record count goes 220 → 190. All records in both versions are `suite: "benchmark"`. Of the 182 records present in both, **zero** differ — no pass/fail flips, no field changes. The entire delta is the rows below. **Removed — the incomplete Opus 4.8 row (4 records).** `claude-code-opus-4.8` and `claude-code-opus-4.8-no-skills`, each holding only `build-cli-001-bootstrap-app` and `build-database-001-migrate-postgres-to-supabase` (both `stage: build`, both passing) — exactly the shape that produces Build 100% / em-dashes / 100% TOTAL. Supporting evidence that this data was accidental: there is no `experiments/claude-code-opus-4.8*.ts` definition in the repo, so no refresh run could have generated it. The only other mention of `claude-opus-4-8` anywhere is a label formatting assertion in `apps/web/src/lib/format.test.ts`. **Removed — results for three new supabase-js SDK evals across the other 14 benchmark rows (34 records).**⚠️ This is the part that goes beyond the reported bug and is worth a second opinion: - `build-auth-001-email-password-flow` - `build-dataapi-001-relational-report` - `build-dataapi-002-restock-alert-report` Affected rows: `claude-code-opus-5`, `claude-code-sonnet-5`, `codex-gpt-5.4-mini`, `codex-gpt-5.6`, `opencode-kimi-k3`, and each of their `-no-skills` counterparts. Their eval *definitions* stay in `evals/` (this PR touches only the JSON), so the next benchmark refresh should repopulate them — they'll just be absent from the site until it runs. **Restored — metadata re-attributions that `#129` introduced (8 records).** Not new data; the same two Build evals moving back to their pre-`#129` identity: | Eval | Currently on `main` | After this PR | |---|---|---| | `build-cli-001-bootstrap-app`, `build-database-001-migrate-postgres-to-supabase` | `claude-code-opus-4.8` / `claude-opus-4-8` | `claude-code-opus-5` / `claude-opus-5` | | same two, `-no-skills` | `claude-code-opus-4.8-no-skills` | `claude-code-opus-5-no-skills` | | same two, codex | `codex-gpt-5.6` @ effort **low** | `codex-gpt-5.6` @ effort **medium** | | same two, codex `-no-skills` | `codex-gpt-5.6-no-skills` @ **low** | `codex-gpt-5.6-no-skills` @ **medium** | ## In-flight refresh — possible conflict The benchmark refresh run started around 16:54 UTC today ([run 31121347612](https://github.com/supabase/evals/actions/runs/31121347612)) had not landed on `main` as of `85c4f6a`. Per `.github/workflows/eval-refresh.yml`, scheduled/manual runs open their own PR on `chore/refresh-eval-results-*` rather than pushing to `main`. If that run produces a refresh PR touching `eval-results.json`, it will conflict with this branch and whichever merges second wins — so it's worth checking for that PR before merging. A refresh should regenerate the three SDK evals and should not reintroduce the Opus 4.8 row. ## Test plan - [ ] Confirm the Benchmark table on a preview build no longer shows a Claude Code / Opus 4.8 row - [ ] Confirm remaining rows' Build/Deploy/Investigate/Resolve/TOTAL figures match pre-`#129` values - [ ] Decide whether losing the three SDK evals' results until the next refresh is acceptable, or whether a forward fix (drop just the Opus 4.8 records and repair the re-attributions) is preferred --- _Generated by [Claude Code](https://claude.ai/code/session_01TXk7jJgnnwPuancUutKkAg)_ Co-authored-by: Claude <noreply@anthropic.com>
2 participants
Adds four evals for supabase-js: dedicated coverage for the SDK's two most-used surfaces (auth and the PostgREST query builder), a regression canary for the API key migration, and a second Data API scenario probing an SDK-adoption gap the first one turned up.
build-auth-001-email-password-flow(benchmark): signup/sign-in/profile flow against stubbed app code. Gates on agents reaching for@supabase/supabase-jsunprompted (thebuild-functions-005pattern); the profile display name only survives if sent as signup user metadata.build-dataapi-001-relational-report(benchmark): backend reporting script over a seeded relational schema (embedded relations, aggregation). Gates on using the SDK query builder via the Data API rather than psql or a raw Postgres driver.build-dataapi-002-restock-alert-report(benchmark): same shape asbuild-dataapi-001— bare prompt, emptypackage.json, no SDK named — but a different schema/domain (inventory restock alerts vs. a sales report). Added after refreshed results onbuild-dataapi-001showed every claude-code variant (4/4) reaching for@supabase/supabase-jsunprompted while every codex variant (0/4) hand-rolled raw HTTP instead, despite otherwise-correct, RLS-safe implementations. One scenario isn't enough to tell a real model tendency from an artifact of that specific prompt, so this companion checks whether the split generalizes.resolve-sdk-001-legacy-key-migration(regression): seeds a working app on the legacy demo JWTs; agents must migrate it tosb_publishable_/sb_secret_keys without breaking behavior or leaking the secret key into client code.All four were sanity-checked locally against a real stack on the pinned CLI (2.109.1): migrations apply, reference solutions pass every scorer check, and the checks discriminate (missing signup metadata, publishable-key impersonation, leftover legacy keys, and SDK vs. raw-HTTP implementations all fail as intended). A scoped
.gitignoreexception ships the key-migration eval's seeded.env, which holds only the well-known local demo keys.Fix along the way:
build-dataapi-001-relational-reportandresolve-sdk-001-legacy-key-migrationinitially failed 100% of agents identically — a harness bug, not an agent gap. Both had omittedgotruefrom theirservices:frontmatter, so the local stack's auth service never started andsupabase statuscouldn't emitPUBLISHABLE_KEY/SECRET_KEY. Fixed by addinggotruetoservices:(matchingresolve-dataapi-002-secure-default-grants's pattern); results refreshed and now produce real signal.build-dataapi-002-restock-alert-reportstill needs a live run (no API keys in this pass) to confirm whether the adoption split holds.