fix: executor experiment config, execution, and scoring - #3
Merged
Conversation
mattrossman
changed the title
mattrossman
marked this pull request as draft
May 18, 2026 18:54
mattrossman
changed the title
mattrossman
marked this pull request as ready for review
May 18, 2026 20:01
barryroodt
added a commit
that referenced
this pull request
Jul 24, 2026
…OW_MISSING_SOURCES
The fail-loud partners fix immediately caught a real, previously silent
failure: the misc API rejects the legacy anon key locally ('Legacy API
keys are disabled'), so partners has ALWAYS quietly indexed as zero on
this machine - strong candidate for the 27 purged rows. Upstream keeps
the hard throw (abort before purge); locally the ALLOW flag now treats
partners like lint-warnings: warn, register '/partners/integrations/%'
in the shared scope registry (moved to base.ts, both sources push), and
skip. Caught by cold run #3.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Earlier testing relied on a global
executorinstall on my machine. This adds it to the package deps in@supabase-evals/coreand resolves itspackage.binentry at runtime to run it in a more portable way.Newer versions of executor don't load sources from
executor.jsonc(see UsefulSoftwareCo/executor#807) which also messed with eval results. I've switched to the CLI interface for adding the OpenAPI source with a temporary data directory that cleans up after tests.Other changes to fix execution and scoring of the
*-executorexperiment:resumecalls before hittingmax_steps.stoppedReasonin result JSON to help debug when model hitsmax_stepsinstead of only showing an emptyagentReport.15%,0.15, or3 errors / 20don’t false-negative due to formatting.