fix(evals): stop comments satisfying an expected API - #509
Merged
Conversation
expected_apis are matched as plain substrings against the whole file, so
an entry like "thumbs" is satisfied by a comment reading "detect a thumbs
up gesture". Code that calls nothing real still collected credit on the
API dimension.
Measured on gestures-thumbs-up with a file calling createGestureDetector,
onThumbsUp and spawnConfetti, none of which exist: api_match 0.333 and
composite 0.778 out of 1.0, earned purely by the word appearing in a
comment. The same file now gets api_match 0.0 and composite 0.667, which
is the floor for anything that parses and avoids the forbidden patterns.
Only comments are removed. String literals stay, because this SDK passes
names as strings and setGestureEnabled('thumbs-up', true) is real usage
rather than prose. Stripping them as well would have marked correct code
wrong. Imports keep matching the raw source either way, since a specifier
like @dimforge/rapier3d only ever appears inside a string.
Tests use stdlib unittest so they need nothing installed. Neutering the
stripper fails six of them.
The docstring also still described the old quarter-per-dimension
composite, which stopped being true when import_match started being
dropped as vacuous.
Four tasks listed a plain word in expected_apis: "thumbs" on the two
gesture tasks, "query" on the two AI ones. Matching is by substring, so
those are satisfied by any mention anywhere, which is a weak signal even
now that comments are excluded.
"thumbs" becomes "thumbs-up", which is the gesture name the SDK actually
uses and what correct code passes to setGestureEnabled or compares
event.detail.name against. Invented code that writes "thumbs up" with a
space no longer counts.
"query" becomes ".query(", so it has to be a call. The canonical form is
xb.ai.query({prompt}), and requiring the member call still matches an
aliased const ai = xb.ai without matching the English word.
On gestures-thumbs-up: a file calling createGestureDetector, onThumbsUp
and spawnConfetti scores 0 of 3 APIs, and one using enableGestures,
setGestureEnabled('thumbs-up') and the gestureRecognition event target
scores 3 of 3.
Numbers from these tasks are no longer comparable with the earlier
published run, which scored some of this credit for prose.
The longer gesture name pushed those lines past the print width.
dli7319
approved these changes
Aug 8, 2026
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A file that calls
createGestureDetector,onThumbsUpandspawnConfetti, none of which exist, scores 0.778 out of 1.0 ongestures-thumbs-up. It gets there because the task lists"thumbs"inexpected_apis, matching is a substring check over the whole file, and the file has a comment reading "detect a thumbs up gesture". So it collected a third of the API dimension for prose.That's the dimension the whole thing exists to measure, and it was being paid out for a comment.
Now only comments are stripped before matching. Same file scores 0.0 on APIs and 0.667 composite, which is just the floor for parsing and not tripping the forbidden patterns.
I did try stripping string literals too, and that was wrong. Gesture names are passed as strings here, so
setGestureEnabled('thumbs-up', true)is real usage and correct code would have started failing. Imports were never going to work that way either, since a specifier like@dimforge/rapier3donly ever appears inside a string.Second commit tightens the four tasks that listed a bare word.
"thumbs"becomes"thumbs-up", which is the actual gesture name, so invented code writing "thumbs up" with a space gets nothing."query"becomes".query("so it has to be a call rather than the English word, and that still matches if someone aliasesconst ai = xb.ai.On
gestures-thumbs-upthe invented file now scores 0 of 3 APIs, and one usingenableGestures,setGestureEnabled('thumbs-up')and thegestureRecognitionevent target scores 3 of 3.Tests are stdlib
unittest, so nothing to install:Neutering the stripper fails six of the nine, so they're actually holding something.
Worth saying plainly: these four tasks won't be comparable with the earlier published run, since that one handed out some of this credit for prose. Probably makes the with-skills vs without-skills gap look bigger than before, since invented code was sitting on a floor closer to 0.78 than 0.67.
Type of Change
Media / Screen Recordings & Screenshots (If Applicable)
Nothing visual, it's the scoring script for the eval prototypes.
Checklist
gestures-thumbs-upbefore and after, plus 9 unit tests on the comment stripper.unittestrather than adding pytest.