test: add bats tests for benchmark shell scripts#85
Conversation
There was a problem hiding this comment.
Pull request overview
Adds bats-core test coverage for the benchmark reporting scripts and wires the test suite into CI to gate release/benchmark workflows.
Changes:
- Add bats tests for
scripts/compare.sh(regressions, thresholds, missing baseline/latest, absolute limits, size regression). - Add bats tests for
scripts/format-release.sh(pivot formatting, deltas, multi-method/tool output, empty input, usage errors). - Add a CI
testjob and makerelease/benchmarkdepend on it.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
tests/format-release.bats |
New bats tests validating markdown formatting and delta rendering for release output. |
tests/compare.bats |
New bats tests validating regression detection behavior and threshold handling. |
.github/workflows/ci.yml |
Introduces a test job and gates release/benchmark on lint + tests. |
| @test "marks new benchmarks without baseline data" { | ||
| make_json "$TMPDIR/baseline.json" '{}' | ||
| make_json "$TMPDIR/latest.json" \ |
There was a problem hiding this comment.
make_json already wraps the provided benchmark entries in { ... } via "benchmarks": {$benchmarks}. Passing '{}' here produces invalid JSON ({{}}). Call make_json with no additional benchmark args (or adjust make_json to accept a pre-wrapped object) when you want an empty benchmarks map.
| @test "absolute threshold triggers FAIL" { | ||
| make_json "$TMPDIR/baseline.json" '{}' | ||
| make_json "$TMPDIR/latest.json" \ |
There was a problem hiding this comment.
This call passes '{}' into make_json, which results in "benchmarks": {{}} (invalid JSON). For an empty baseline, invoke make_json "$TMPDIR/baseline.json" with no benchmark args (or update make_json to avoid double-wrapping).
| @test "absolute threshold passes when under limit" { | ||
| make_json "$TMPDIR/baseline.json" '{}' | ||
| make_json "$TMPDIR/latest.json" \ |
There was a problem hiding this comment.
This call passes '{}' into make_json, which results in "benchmarks": {{}} (invalid JSON). Use make_json "$TMPDIR/baseline.json" with no additional args (or make make_json accept a raw JSON object) to produce an empty benchmarks map.
| - uses: actions/checkout@v6 | ||
| - name: Install bats | ||
| run: sudo apt-get install -y bats | ||
| - name: Run tests | ||
| run: bats tests/ |
There was a problem hiding this comment.
The bats tests invoke scripts/compare.sh and scripts/format-release.sh, both of which hard-require jq and exit non-zero if it’s missing. The test job should install jq explicitly (or add a dedicated setup step) to keep CI deterministic across runner image changes.
Summary
compare.shandformat-release.shtestjob to CI pipeline, gating bothreleaseandbenchmarkjobsCloses #48