fix(benchmarks): parse both v1 and v2 BENCHMARK.md formats, and unify agent identifiers - #400
Merged
Merged
Conversation
SkillEvaluator 0.9.x emits a redesigned BENCHMARK.md. The aggregator was
written against the older layout, so every re-signed skill silently lost
its evaluation data in benchmarks.json: has_results went false, with null
tasks, no agents and no uplift, even though the underlying report carried
richer results than before.
Both layouts are now accepted, since the catalog will hold a mix until
every skill has been re-signed:
- Evaluation Summary (v1) and Evaluation Metadata (v2) field lists
- "Agents Used" bullets (v1) and the inline "- Agents:" line (v2)
- "## Results" and "## Results at a Glance" tables, keyed on either
Dimension or Measure
- v1 cells ("100% (+70%)") and v2 cells ("45% -> 98% (+53 points)"),
recording the skill-assisted score in both so the series stays
comparable
- agent names normalised, so v2's "Claude Code (`provider/model`)" and
v1's `claude-code` aggregate as one agent
Also fixes a verdict misread. In v2 the real verdict is a callout above
the report body, and the old pattern instead matched the methodology
bullet "- Overall verdict: PASS only when every configured dimension
passes...", which appears in every v2 report. Any failing v2 skill would
have been recorded as PASS. The v1 pattern is now anchored to
end-of-line, and the v2 callout is matched explicitly.
Effect on the current catalog: 5 skills regain results (skills without
results 27 -> 22) and result rows go 2955 -> 3000. Only the 7 skills
already carrying v2 reports change; v1 records are untouched.
Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Holding this pending BI review — see the agent-identifier section. Not for merge yet.
Problem 1: re-signed skills drop out of the feed
SkillEvaluator 0.9.x emits a redesigned
BENCHMARK.md.aggregate_benchmarks.pywas written against the older layout, so every skill that re-signs silently loses its evaluation data:has_resultsgoes false, with null tasks, no agents and no uplift, even though the new report carries richer data than the one it replaced.Two skills re-signed this week and both lost their numbers:
nemo-automodel-distributed-traininghas_results: falsenemo-automodel-model-onboardinghas_results: false7 of 323 reports are v2 today. With 21 skills still to re-sign, the feed behind the adoption dashboard would empty out as the re-sign campaign succeeded.
Problem 2: every v2 report parsed as PASS
In v2 the verdict is a callout above the report body. The old pattern didn't match it, and instead matched the methodology bullet present in every v2 report:
So a failing v2 skill would have been recorded as
PASS. The v1 pattern is now anchored to end-of-line, and the v2 callout is matched explicitly.Problem 3: the feed carries four identifiers for two agents
Independent of the format work, the results rows today contain:
claude-codecodexClaude Code (\aws/anthropic/bedrock-claude-opus-4-8`)`Codex (\openai/openai/gpt-5.5`)`67 skills were signed by an evaluator version that wrote the model ID into the results-table header. Anything grouping the feed by agent sees four agents, so per-agent aggregates split and undercount.
This PR normalises all four to
claude-codeandcodex(1500 rows each). This rewrites 665 existing rows across 67 skills and is the change that needs BI sign-off — it is a data-shape change to a field the dashboard joins on, not just a parser fix.What changed
## Agents Usedbullets (v1) and the inline- Agents:line (v2)## Resultsand## Results at a Glancetables, keyed onDimensionorMeasure100% (+70%)) and v2 cells (45% → 98% (+53 points)), recording the skill-assisted score in both so the series stays comparableOverallrow skipped, soaverage_uplift_pctstays comparable with v1Verification (local)
skills_without_results27 → 22;result_row_count2955 → 3000schema_versionstill 2aggregate_benchmarks.py --checkpasses against the committed file🤖 Generated with Claude Code