Bench/genui formats - #62
Merged
Merged
Conversation
The benchmark tests AGenUI/AGenUI, the native renderer SDK implementing the A2UI v0.9 protocol — not a2ui-project/a2ui, the protocol project itself, which was not tested. Labelling the column "A2UI" attributed AGenUI's results, and the finding that no injectable prompt is shipped, to the protocol. Renames the adapter label and every reference in the report and README. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fourth column tested AGenUI, a third-party renderer SDK. This adds a2ui-project/a2ui itself — the protocol project google/A2UI redirects to — as a fifth format, prompted by its own a2ui_agent TransportFormat prompt generator against the standard v0.9 basic catalog, and validated by its own TransportParser.parse_response(). 270 generations across the same three rungs. The two are not interchangeable, which is why both are kept: at the shared 8k ceiling AGenUI truncates 52.2% of Opus 5 generations against A2UI's 7.8%, so treating AGenUI as a proxy would have badly misrepresented the protocol. Prompt config is include_schema=True, include_examples=False (~10.1k tokens). With neither flag the prompt is 213 tokens and describes no components; with examples it is ~49.6k, four times any other prompt here. Also fixes two harness bugs this exposed: - generate() only caught fetch rejections, not failures while streaming the response body, so an ECONNRESET inside res.json() killed a 270-generation run. The whole attempt now sits inside the retry, with 5xx/429 retried and 4xx failing fast. - the report's completeness filter demanded an exact 90 generations per format and so silently dropped a fully-run model over 2 network errors; it now allows 90% of the expected sweep. Existing ledger records were migrated from format "a2ui" to "agenui". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script filtered format === 'a2ui', which after the A2UI/AGenUI split selects the new A2UI generations while validating them with AGenUI's validator — so re-running it would have produced nonsense. Renamed to crosscheck-agenui.mts and retargeted at the agenui arm. The figures in the report were unaffected and still reproduce exactly: 100.0/84.2, 81.1/70.0, 93.3/12.2 over the same 218 non-truncated generations. A2UI needs no equivalent cross-check — it is already validated by its own parser. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were transient "operation was aborted" timeouts on chart/minimal and chart/realistic that exhausted the retry budget. Re-running succeeded, so all three models now have the full 90 A2UI generations each. The superseded error records are pruned from the ledger — an error record for a cell that later succeeded would otherwise make the report state API errors for generations that are present and scored. Error records with no successful counterpart would be kept; there are none. Gemma A2UI renderable rate moves 80.7% -> 81.1% now that the two cells are scored rather than skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
benchmarks/*/results/** is generated output (results.json, REPORT.md, the ledger) and benchmarks/*/vendor/** is third-party source vendored verbatim — reformatting the latter would contradict vendor/NOTICE.md, which states every vendored file is byte-identical to upstream. Follows the existing evals/results*.json precedent. All benchmark sources pass biome format and lint; the 14 remaining repo-wide failures are pre-existing in demo/ and packages/*/package.json and untouched by this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
barlipdev
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
added A2UI and changed AGenUI
Research output only — no published package is touched.
Type of Change
Packages Affected
None.
benchmarks/genui-formatsisprivate: trueand never published. Repo-level changes arepnpm-workspace.yaml(+1 line),pnpm-lock.yaml, and twobiome.jsonignore entries.Checklist
biome formatandlintpass on all 18 benchmarksources. (14 repo-wide failures in
demo/andpackages/*/package.jsonare pre-existing onmainand untouched here.)pnpm verifyis the suite: 5 gates × 5 adapters (accepts ownfixture, rejects own corrupted fixture, rejects all four foreign formats, rejects empty/prose,
no format hints in user prompts).
pnpm verifygreen;pnpm scorereproduces every number offline.How to Test
cd benchmarks/genui-formats && pnpm verify— proves each validator accepts its own format andrejects the other four. Prints
All adapter checks passed.pnpm score— re-scores the committed 1,377 generations offline. Free, no API key.pnpm report— regeneratesresults/REPORT.md; should match the committed file.OPENROUTER_API_KEY):pnpm bench --models google/gemma-4-26b-a4b-it --scenarios contact-form/minimal --k 1(~$0.01).Results
"Every time" rate — share of scenarios where all 5 repeats rendered:
Full write-up:
benchmarks/genui-formats/results/REPORT.md.