Skip to content

Fix/benchmark - #63

Merged
gitsad merged 2 commits into
mainfrom
fix/benchmark
Aug 10, 2026
Merged

Fix/benchmark#63
gitsad merged 2 commits into
mainfrom
fix/benchmark

Conversation

@gitsad

@gitsad gitsad commented Aug 10, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Corrects the OpenUI arm of the generative-UI format benchmark. Their published
benchmarks/system-prompt.txt is byte-identical to their repository but no longer matches the
library they ship: openuiLibrary.prompt() from @openuidev/react-ui@0.13.5 emits 17,431 chars
against the artifact's 13,080, drops ## Examples, adds ## Action - Button Behavior and
## Final Verification, and introduces the Modal and Action components. We were benchmarking
a prompt no integrator receives, so the scored arm now uses the shipped library prompt with the
schema regenerated from the same version.

All 270 generations were re-run rather than adjusted after the fact. Also pins the A2UI Python
setup, which lived in a /tmp venv the OS cleans, leaving the committed benchmark unscoreable on
a clean machine.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Packages Affected

None. Only benchmarks/genui-formats (private, never published) and pnpm-lock.yaml.

Checklist

  • I have read the CONTRIBUTING guide.
  • My code follows the existing code style (biome format and lint pass on all 19 files).
  • I have added or updated tests that cover my changes — pnpm verify gained probe-adapter
    handling so the superseded arm skips the fixture and cross-validation checks that cannot
    apply to it.
  • All tests pass — pnpm verify green; pnpm score reproduces every number offline.
  • Type-checking — the benchmark package has no tsconfig.json, so pnpm typecheck does not
    cover it. tsx compiles it on every run and pnpm verify exercises all six adapters.
  • Changeset — not needed, no published package affected.
  • MDMA schemas — n/a.
  • Sensitive fields — n/a.

How to Test

  1. cd benchmarks/genui-formats && ./a2ui-python/setup.sh — installs the A2UI SDK from its pinned
    commit. Required before scoring, and the thing this PR makes reproducible.
  2. pnpm verify — all six adapters, five gates. Prints All adapter checks passed.
  3. pnpm score && pnpm report — re-scores the committed 1,647 generations offline and regenerates
    results/REPORT.md. Free, no API key.
  4. Optional, needs OPENROUTER_API_KEY:
    pnpm bench --formats openui-v2 --models google/gemma-4-26b-a4b-it --scenarios contact-form/minimal --k 1 (~$0.01).

Results

"Every time" rate, five repeats per scenario:

Format Opus 5 GPT-5.6-terra Gemma-4-26B
MDMA 94.4% 83.3% 94.4%
OpenUI Lang 94.4% 94.4% 50.0%
json-render 83.3% 72.2% 38.9%
A2UI (transport) 83.3% 61.1% 44.4%
AGenUI 38.9% 72.2% 77.8%

Effect of the correction on OpenUI:

Model Published artifact Shipped library Delta
Opus 5 83.3% 94.4% +11.1pp
GPT-5.6-terra 100.0% 94.4% -5.6pp
Gemma-4-26B 55.6% 50.0% -5.6pp

Reviewers should know this correction did not favour us. It gained OpenUI 2 scenarios on the
flagship and cost them 1 on each cheaper model, averaging to no change across the ladder, and
OpenUI now ties MDMA at 94.4% on Opus 5 where we previously showed them at 83.3%. The published
artifact was not systematically unfair to them; it scored better on two rungs of three.

The finding is written up as section 8 of results/REPORT.md, including the prompt diff and the
before/after comparison.

gitsad added 2 commits August 10, 2026 12:35
Its SDK lived in a /tmp venv the OS cleans, leaving the committed benchmark unscoreable on a clean machine.
Their published benchmarks/system-prompt.txt no longer matches @openuidev/react-ui@0.13.5, so we were benchmarking a prompt no integrator receives.
@gitsad
gitsad merged commit 1f60d37 into main Aug 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants