refactor(ci): run geth benchmark CI only on benchmarks/** branches - #3249
Merged
spencer-tb merged 3 commits intoJul 29, 2026
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## forks/amsterdam #3249 +/- ##
================================================
Coverage 93.41% 93.41%
================================================
Files 625 625
Lines 37035 37035
Branches 3386 3386
================================================
Hits 34597 34597
Misses 1668 1668
Partials 770 770
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
spencer-tb
approved these changes
Jul 29, 2026
spencer-tb
left a comment
Contributor
There was a problem hiding this comment.
LGTM!
cc @danceratopz for when you are back
21 tasks
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Benchmark CI previously coupled an external, spec-matched geth dependency into checks that ran on (nearly) every PR:
benchmark.yaml'sbench-gasjob fills fixtures with geth and re-verifies them against EELS, so whenever the EELS spec moved ahead of the pinned geth ref, unrelated spec PRs failed the benchmark job (see #3214: EELS on devnet-8 vs geth onglamsterdam-devnet-7).This PR separates benchmarking so that mainline development is never blocked by an external client, while the geth-backed work moves to dedicated
benchmarks/**branches:test_benchmarking.pydefaults to EELS. The benchmark-plugin fills previously hardcoded--evm-bin(defaulting to geth'sevm). They now omit--evm-binwhenEVM_BINis unset, falling through tofill's in-repo EELS t8n; settingEVM_BINstill targets a specific binary. This removes the only reason those tests needed an external client.--ignore=.../test_benchmarking.pyis removed fromjust test-tests, so the benchmark-plugin tests now run as part of the standard unit tests on everyforks/**PR viatest.yaml— on EELS, with no geth. The redundanttest-tests-benchrecipe is deleted.test.yamlbehaves differently onbenchmarks/**. On those branches it builds the pinned geth and setsEVM_BIN, so the same benchmark tests run against a real client; on every other branch the build step is skipped and the tests use EELS. (EVM_BINis read only bytest_benchmarking.py, so this has no effect on the rest of the suite.)benchmark.yamlruns only onbenchmarks/**. It now holds the geth-backed sanity checks (bench-gascross-check,bench-opcode,bench-opcode-config) and the fixture-artifact build. These branches pin a spec version together with a matching geth ref, so the geth↔EELS cross-check is expected to agree there.Net effect:
forks/**PRs get full benchmark-plugin coverage on EELS with no external dependency, and the geth cross-check/fills run where geth and the spec are deliberately kept in sync. This is the first step of a larger benchmarking separation; creating thebenchmarks/amsterdambranch and the release/publish flow are follow-ups.Files changed:
packages/testing/.../filler/tests/test_benchmarking.py— EELS-by-default_evm_bin_args()helper..github/workflows/test.yaml— addbenchmarks/**; conditional geth build +EVM_BINintest-tests..github/workflows/benchmark.yaml— scope tobenchmarks/**; dropunit-tests(now covered bytest.yaml).Justfile— un-ignore the benchmark tests intest-tests; deletetest-tests-bench.docs/getting_started/verifying_changes.md— update the benchmark row.Related Issues or PRs
#3244
Checklist
just static<type>(<area>): <title>, where<type>and<area>come from an appropriateC-<type>, respectivelyA-<area>, label. The title should match the target squash commit message.Cute Animal Picture