fix(evals): correct false-negative scoring (seed dir, endpoint case, brittle test check) - #50
Merged
Merged
Conversation
…brittle test check) + status retry
Audit of Claude Code eval runs surfaced false negatives that mis-score any agent:
P1 — seed never applied. investigate-auth-001 and investigate-realtime-001
shipped their seed under `seed/`, but the harness reads hosted state from
`remote/` — so both ran against an empty DB (scorer threw `relation "profiles"
does not exist`; the realtime judge penalized the agent for rebuilding the
scenario it had to recreate). Rename `seed/` → `remote/` for both.
P2a — CLI endpoint case. build-cli-002's "supabase db diff used" check matched
`tc.endpoint === "bash"`, but CLI agents (Claude Code) emit `Bash`, making the
check unwinnable for them. Compare case-insensitively.
P2b — `supabase status` readiness. discoverStackConfig() read the API keys once
and threw if absent, false-failing getClient()-based REST checks when gotrue
hadn't finished coming up. Retry briefly (5×2s) before failing.
build-tests-001 — the "negative tests catch the bug (failedTests > 0)" check
hard-coded one test style (assert correct isolation → fails on the buggy posts
policy) and contradicted the diagnosis judge: an agent that writes passing
characterization tests and correctly diagnoses `posts` (judge passes) still
failed this check. Drop it; the judge already requires identifying `posts` from
the results. Keep files-exist + "tests ran and pass"; de-style the judge wording
("test failures" → "test results").
Not changed: edge-runtime npm-import rejection (needs confirmation that the real
runtime accepts pure-JS imports — the mock may be wrong) and the deliberately
strict RLS judge in resolve-dataapi-001.
sandbox/framework typecheck clean; sandbox tests 24/24.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
…e:/esm.sh)
The real Supabase Edge runtime (Deno) accepts npm:/jsr:/node:/esm.sh and bare
imports, but the mock's require shim only resolved @supabase/supabase-js and
threw "edge function import not supported" for everything else — false-failing
any function that used, e.g., npm:jwt-decode or node:crypto (build-functions-004).
Resolve node-like specifiers against the eval runtime's modules (createRequire):
- node: builtins, npm:/jsr: prefixes, esm.sh/CDN URLs, and bare specifiers, with
the version stripped (pkg@1.2.3/sub -> pkg/sub).
- @supabase/supabase-js keeps its runtime-fetch shim.
- Node can't fetch on demand like Deno, so a package an eval needs must be
installed; an unresolvable one now errors clearly ("not available in the eval
runtime") instead of "import not supported".
- Bumped the vm top-level timeout 100ms→1000ms (require() now loads real packages).
Verified end-to-end via bootPlatformBackend: a function importing npm:zod +
node:crypto returns 200 (previously threw). Confirmed in a temp dir that
npm:jwt-decode resolves+runs too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mattrossman
approved these changes
Jun 19, 2026
mattrossman
left a comment
mattrossman
left a comment
Collaborator
There was a problem hiding this comment.
Down the road we may want to rethink the edge function JS imports stuff, support for @supabase/supabase-js was a short term hack and I wonder how far we can get w/ this solution of installing everything the agent might need in our evals repo deps. Maybe there's a middle ground between that and spinning up the full edge functions runtime in local stack mode, like running actual Deno + letting it resolve deps with any version.
Contributor
Author
Answered here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes false-negative eval scoring found while auditing claude-code-haiku-4.5 and claude-code-sonnet-4.6 runs.
Fixed
investigate-auth-001-deleted-user-accessandinvestigate-realtime-001-subscribed-no-eventsbecause their seed was still under the oldseed/driectory (renamed toremote/. So both ran against an empty DB (scorer threwrelation "profiles" does not exist; the realtime judge penalized the agent for rebuilding the scenario it was forced to recreate).build-cli-002's "supabase db diff used" check matchedtc.endpoint === "bash", but Claude Code emit"Bash"instead. Lowercasedtc.endpointdiscoverStackConfig()read the API keys once and threw if absent. Now retries briefly (5×2s) before failing.postsstill failed it, because the "negative tests catch the bug (failedTests > 0)" check hard-coded one test style contradicted the diagnosis judge. Dropped it and kept only the judge assessment.requireshim only resolved@supabase/supabase-jsand threw "import not supported" fornpm:/node:/esm.shimports the real Deno runtime accepts (build-functions-004usednpm:jwt-decode). Now resolvesnode:builtins,npm:/jsr:/CDN URLs/bare specifiers (version-stripped) against the runtime's modules; unresolvable packages error clearly instead of being rejected. Verified viabootPlatformBackend(npm:zod+node:crypto→ 200).Notes
These are framework/eval-definition fixes, independent of the Claude Code feature work (#47).