Let /benchmark compare against another framework#1031
Merged
Conversation
Closes #741. The delta table always baselined a framework against its own published results, which answers "did my change help?". When you are developing a tuned build or a successor entry, the useful question is usually "how does this compare to the entry it derives from?", and there was no way to ask it. /benchmark -f genhttp-11 --compare genhttp ./scripts/compare.sh genhttp-11 --compare genhttp compare.sh gains --compare <framework>: it resolves that framework's display_name from its meta.json and looks up that name in site/data instead of its own. The rest of the table is unchanged. Three details that matter for reading the result: The output states the baseline in a line above the tables, because a raw "+6361%" against a different framework would otherwise be indistinguishable from an enormous regression against the framework's own history. Profiles the comparison framework does not run report n/a rather than NEW. NEW means "this framework has no published result yet"; against a different entry it would read as though the profile were new, when it simply isn't run there. A framework directory that doesn't exist is reported ("No such framework to compare against: `nope`") instead of silently comparing against nothing, which would have shown NEW for every cell. Plumbing: pr-commands.yml parses --compare <fw> (and --compare=<fw>) from the comment and forwards it; benchmark-pr.yml takes it as an input, passes it to compare.sh, and records it in the comment header as "Compared to: `genhttp`". The table format is untouched, so the existing parser that lifts deltas out of compare.sh's output keeps working. The flag is documented in compare.md, the README command table and the pull request template. Verified against real published data: with no flag the output is byte -identical to before, --compare genhttp reprices every delta against genhttp's 33,073 rps baseline instead of genhttp-11's 1,997,039, the typo and no-results paths print their messages, and both workflow files still parse as YAML. Comment parsing checked across flag orders and the --compare=<fw> form.
Adding --compare put a fourth comment-derived value into a shell script via
${{ }} interpolation, which is how GitHub Actions script injection works. The
value is parsed out of a PR comment with grep -oP '\S+', and \S includes
quotes and semicolons, so a comment like
/benchmark -f x --compare a";touch${IFS}/tmp/PWNED;#
expands inside pr-commands.yml to
gh workflow run benchmark-pr.yml ... -f compare="a";touch${IFS}/tmp/PWNED;#"
and the runner executes it. Confirmed locally by reproducing that expansion.
This matters because handle-benchmark has no approval gate: it fires on any
issue_comment containing /benchmark, from any user, with actions: write and a
GITHUB_TOKEN. The environment: runner gate protects the later benchmark job in
benchmark-pr.yml, not this one.
framework, profile and save were already interpolated the same way, so this
closes all four rather than only the one I added.
Two layers:
Sanitize at the source. Framework directories and profile names are
[A-Za-z0-9._-]; anything else becomes empty, so a crafted value is dropped
before it reaches any script. Verified: genhttp, genhttp-11 and
aspnet-minimal_nginx pass; the payload above, $(whoami), "a b" and
../../etc/passwd all become empty, and the run falls back to the normal
self-comparison instead of failing.
Pass through env. The dispatch step and the compare step now receive values
as environment variables instead of having them pasted into the script text,
so nothing in them can be parsed as code even if the whitelist is ever
loosened. compare_args also becomes an array, so the argument is passed
whole rather than relying on word splitting.
Functionally unchanged: the full chain still parses
"/benchmark -f genhttp-11 -t baseline --compare genhttp" correctly, produces
the same table, and the delta parser in the post-results step still reads it.
The command tables listed the flag but the text under them still said results are "automatically compared against the current leaderboard", which no longer describes what happens when --compare is used. Both now state what the deltas are measured against by default, and show the flag in context. The PR template was also missing the `-t <test> --save` combination that the README already documented; both tables now list the same five forms.
MDA2AV
added a commit
to Kaliumhexacyanoferrat/HttpArena
that referenced
this pull request
Jul 24, 2026
Six merges landed since the last sync: MDA2AV#1030, MDA2AV#1031, MDA2AV#1032, MDA2AV#1033, MDA2AV#1034 and MDA2AV#1036. One real conflict, in the composite blurb, where both sides had changed the same line for different reasons: main (MDA2AV#1032) made the text state which normalization rule is active, so a screenshot can't be ambiguous about whether scores were rescaled to the current filter this branch pointed the "How it works" link at /docs/scoring/composite -score/ instead of the hash route Both are kept: MDA2AV#1032's wording with this branch's real URL. Everything else merged on its own, but the two structural ones were checked rather than assumed: MDA2AV#1034 moved results from 52 shared per-profile arrays to one file per framework. Git applied that to the renamed generator correctly — it still loads site/data/results/*.json and still emits 1941 rows, and compare.sh kept both its per-framework read and the --compare flag from MDA2AV#1031. MDA2AV#1030 added three echo-ws-limited doc pages after this branch's SEO pass, so they were written with seo_title/description already; all 126 pages now carry both. Generator output on the merged tree: 1941 rows, 126 static doc pages, a 126 -page search index and a 127-URL sitemap. All six workflows validate against a loader that rejects duplicate keys, which is what MDA2AV#1036 had to fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #741.
The delta table always baselined a framework against its own published results, which answers "did my change help?". When developing a tuned build or a successor entry, the useful question is usually "how does this compare to the entry it derives from?" — and there was no way to ask it.
compare.shgains--compare <framework>: it resolves that framework'sdisplay_namefrom itsmeta.jsonand looks that up insite/datainstead of its own. The table itself is unchanged.Three details that matter for reading the result
+6361%against a different framework would otherwise be indistinguishable from an enormous regression against the framework's own history.n/a, notNEW.NEWmeans "no published result yet"; against a different entry it would read as though the profile were new, when it simply isn't run there.NEWin every cell and looks like a successful comparison.Plumbing
pr-commands.ymlparses--compare <fw>(and--compare=<fw>) and forwards it;benchmark-pr.ymltakes it as an input, passes it tocompare.sh, and records it in the comment header as Compared to:genhttp. The table format is untouched, so the existing parser that lifts deltas out ofcompare.shoutput keeps working.Documented in
compare.md, the README command table and the pull request template.Verification
--compare genhttpn/a--compare=<fw>formDeliberately not done
The issue floated "or maybe even both" (deltas against the framework's own results and another's). That means three cells per connection count, and
benchmark-pr.yml's comment parser indexes cells two-per-conn — so it would need changing in lockstep. Happy to add it as a follow-up if you want both baselines side by side.🤖 Generated with Claude Code