Skip to content

fix(evals): stop comments satisfying an expected API - #509

Merged
dli7319 merged 4 commits into
google:mainfrom
salmanmkc:evals-scoring-floor
Aug 8, 2026
Merged

fix(evals): stop comments satisfying an expected API#509
dli7319 merged 4 commits into
google:mainfrom
salmanmkc:evals-scoring-floor

Conversation

@salmanmkc

Copy link
Copy Markdown
Contributor

Description

A file that calls createGestureDetector, onThumbsUp and spawnConfetti, none of which exist, scores 0.778 out of 1.0 on gestures-thumbs-up. It gets there because the task lists "thumbs" in expected_apis, matching is a substring check over the whole file, and the file has a comment reading "detect a thumbs up gesture". So it collected a third of the API dimension for prose.

That's the dimension the whole thing exists to measure, and it was being paid out for a comment.

Now only comments are stripped before matching. Same file scores 0.0 on APIs and 0.667 composite, which is just the floor for parsing and not tripping the forbidden patterns.

I did try stripping string literals too, and that was wrong. Gesture names are passed as strings here, so setGestureEnabled('thumbs-up', true) is real usage and correct code would have started failing. Imports were never going to work that way either, since a specifier like @dimforge/rapier3d only ever appears inside a string.

Second commit tightens the four tasks that listed a bare word. "thumbs" becomes "thumbs-up", which is the actual gesture name, so invented code writing "thumbs up" with a space gets nothing. "query" becomes ".query(" so it has to be a call rather than the English word, and that still matches if someone aliases const ai = xb.ai.

On gestures-thumbs-up the invented file now scores 0 of 3 APIs, and one using enableGestures, setGestureEnabled('thumbs-up') and the gestureRecognition event target scores 3 of 3.

Tests are stdlib unittest, so nothing to install:

python -m unittest discover -s evals/prototypes -p 'test_*.py'

Neutering the stripper fails six of the nine, so they're actually holding something.

Worth saying plainly: these four tasks won't be comparable with the earlier published run, since that one handed out some of this credit for prose. Probably makes the with-skills vs without-skills gap look bigger than before, since invented code was sitting on a floor closer to 0.78 than 0.67.

Type of Change

  • Bug fix
  • New feature / enhancement
  • New demo or sample
  • Documentation update

Media / Screen Recordings & Screenshots (If Applicable)

Nothing visual, it's the scoring script for the eval prototypes.

Checklist

  • Tested in simulator & device: Doesn't touch the SDK or anything that runs in XR. Verified by scoring an invented file and a correct one against gestures-thumbs-up before and after, plus 9 unit tests on the comment stripper.
  • Large Assets ($\ge$ 1MB): None.
  • SDK Dynamic Dependencies: No new dependencies, the tests use stdlib unittest rather than adding pytest.
  • Security: No keys or secrets.

expected_apis are matched as plain substrings against the whole file, so
an entry like "thumbs" is satisfied by a comment reading "detect a thumbs
up gesture". Code that calls nothing real still collected credit on the
API dimension.

Measured on gestures-thumbs-up with a file calling createGestureDetector,
onThumbsUp and spawnConfetti, none of which exist: api_match 0.333 and
composite 0.778 out of 1.0, earned purely by the word appearing in a
comment. The same file now gets api_match 0.0 and composite 0.667, which
is the floor for anything that parses and avoids the forbidden patterns.

Only comments are removed. String literals stay, because this SDK passes
names as strings and setGestureEnabled('thumbs-up', true) is real usage
rather than prose. Stripping them as well would have marked correct code
wrong. Imports keep matching the raw source either way, since a specifier
like @dimforge/rapier3d only ever appears inside a string.

Tests use stdlib unittest so they need nothing installed. Neutering the
stripper fails six of them.

The docstring also still described the old quarter-per-dimension
composite, which stopped being true when import_match started being
dropped as vacuous.
Four tasks listed a plain word in expected_apis: "thumbs" on the two
gesture tasks, "query" on the two AI ones. Matching is by substring, so
those are satisfied by any mention anywhere, which is a weak signal even
now that comments are excluded.

"thumbs" becomes "thumbs-up", which is the gesture name the SDK actually
uses and what correct code passes to setGestureEnabled or compares
event.detail.name against. Invented code that writes "thumbs up" with a
space no longer counts.

"query" becomes ".query(", so it has to be a call. The canonical form is
xb.ai.query({prompt}), and requiring the member call still matches an
aliased const ai = xb.ai without matching the English word.

On gestures-thumbs-up: a file calling createGestureDetector, onThumbsUp
and spawnConfetti scores 0 of 3 APIs, and one using enableGestures,
setGestureEnabled('thumbs-up') and the gestureRecognition event target
scores 3 of 3.

Numbers from these tasks are no longer comparable with the earlier
published run, which scored some of this credit for prose.
The longer gesture name pushed those lines past the print width.
@dli7319
dli7319 merged commit c052245 into google:main Aug 8, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants