ci: add CodSpeed continuous performance measurement#32
Merged
Conversation
Add a pytest-codspeed benchmark suite (simulation mode) covering skit's CPU-bound hot paths and a GitHub Actions workflow that reports results to CodSpeed. - benchmarks/codspeed/: analyzer parse (python/shell/js/ts across 20/200/2000 lines) and store read paths (list_entries, resolve), isolated via SKIT_*_DIR - .github/workflows/codspeed.yml: OIDC-authenticated CodSpeed run on ubuntu-latest - README: CodSpeed badge - pytest-codspeed added to the dev dependency group
Contributor
Author
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The benchmarks landed in benchmarks/codspeed/, which is the right home — and which put them inside two contracts written for the pyperf pipeline. The A/B harness import surface (runtime deps + pyperf + stdlib) exists because benchmark-compare runs benchmarks/ under an older side's venv; a dev-only import there kills every A/B run at startup. The CodSpeed modules are pytest files that harness never imports (`python -m benchmarks run` reaches benchmarks.suites.*), so they get an explicit, reasoned exemption for exactly pytest and its plugin — not a widened rule. Coverage measures the whole benchmarks package, while testpaths keeps the normal pytest run inside tests/, so these files would have been collected by nothing and reported at 0%, taking the 100% floor with them. They are benchmark subjects, like fixtures/noop.py, and are omitted with the same reasoning. Also: the runner label is pinned like benchmark.yml's — simulation mode counts instructions rather than time, so the CPU is irrelevant, but the image decides which library code those instructions run through, and -latest rolling to a new release would land as a step change that reads exactly like a regression. And cancel-in-progress now applies to pull requests only: cancelling a main push would leave a hole in the very baseline every later comparison is measured against.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sets up CodSpeed for continuous performance measurement on skit. This adds a
pytest-codspeedbenchmark suite in CPU simulation mode and a GitHub Actions workflow that runs it on every push tomainand on pull requests, so performance changes are tracked and surfaced directly on PRs.This integration is intentionally separate from the existing
benchmarks/pyperf pipeline and its budget contract — it does not touch or replace them. CodSpeed's simulation instrument provides deterministic, hardware-agnostic CPU measurements (low variance, per-benchmark flamegraphs) that complement the wall-time/footprint numbers the pyperf pipeline produces.What was added
benchmarks/codspeed/— apytest-codspeedsuite targeting skit's CPU-bound hot paths:test_bench_analyzers.py— warmanalyze()for python / shell / js / ts across 20 / 200 / 2000-line sources. Inputs are produced by the existing seeded generator (benchmarks/fixtures/sources.py), so they carry the real constructs each analyzer inspects (parameters, env-defaults, argument parsing).test_bench_store.py—store.list_entries()andstore.resolve()(first/last) over a 200-entry library built through the public store API, isolated viaSKIT_*_DIR(the same isolation the test suite uses — no real library is touched)..github/workflows/codspeed.yml— aCodSpeedworkflow (ubuntu-latest) using OIDC authentication (id-token: write) andmode: simulation. It follows the repo's existing CI conventions: hash-pinned actions,permissions: {}by default,persist-credentials: false,PYTHONUTF8, and path-scoped triggers. Advisory by policy — path-filtered, so it is not intended as a required status check.pyproject.toml/uv.lock—pytest-codspeedadded to thedevdependency group; aper-file-ignoresentry lets the benchmark tests useassert(mirroring thetests/**rule).Verification
codspeed run --mode simulation -- uv run pytest benchmarks/codspeed/ --codspeed— all 15 benchmarks were measured and uploaded, and the run linked tot41372/skit.ruff check,ruff format --check,ty check, andzizmor(workflow audit).benchmarks/and is excluded fromtestpaths = ["tests"], so it does not run in the regularpytestjob or affect the 100% coverage gate. The main suite still collects cleanly (5249 tests).Next steps
main. CodSpeed will start reporting per-PR comparisons on subsequent PRs.