Skip to content

test: add bats tests for benchmark shell scripts#85

Merged
BryanFRD merged 1 commit intomainfrom
test/shell-script-tests
Apr 10, 2026
Merged

test: add bats tests for benchmark shell scripts#85
BryanFRD merged 1 commit intomainfrom
test/shell-script-tests

Conversation

@BryanFRD
Copy link
Copy Markdown
Contributor

Summary

  • Add 19 bats-core tests covering compare.sh and format-release.sh
  • Tests cover: regression detection, threshold comparisons, edge cases (missing baseline, empty results, zero values), delta calculations, pivot table output, multi-tool/multi-method handling
  • Add test job to CI pipeline, gating both release and benchmark jobs

Closes #48

Copilot AI review requested due to automatic review settings April 10, 2026 18:20
@BryanFRD BryanFRD merged commit 736629f into main Apr 10, 2026
9 checks passed
@BryanFRD BryanFRD deleted the test/shell-script-tests branch April 10, 2026 18:21
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds bats-core test coverage for the benchmark reporting scripts and wires the test suite into CI to gate release/benchmark workflows.

Changes:

  • Add bats tests for scripts/compare.sh (regressions, thresholds, missing baseline/latest, absolute limits, size regression).
  • Add bats tests for scripts/format-release.sh (pivot formatting, deltas, multi-method/tool output, empty input, usage errors).
  • Add a CI test job and make release/benchmark depend on it.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
tests/format-release.bats New bats tests validating markdown formatting and delta rendering for release output.
tests/compare.bats New bats tests validating regression detection behavior and threshold handling.
.github/workflows/ci.yml Introduces a test job and gates release/benchmark on lint + tests.

Comment on lines +67 to +69
@test "marks new benchmarks without baseline data" {
make_json "$TMPDIR/baseline.json" '{}'
make_json "$TMPDIR/latest.json" \
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make_json already wraps the provided benchmark entries in { ... } via "benchmarks": {$benchmarks}. Passing '{}' here produces invalid JSON ({{}}). Call make_json with no additional benchmark args (or adjust make_json to accept a pre-wrapped object) when you want an empty benchmarks map.

Copilot uses AI. Check for mistakes.
Comment on lines +90 to +92
@test "absolute threshold triggers FAIL" {
make_json "$TMPDIR/baseline.json" '{}'
make_json "$TMPDIR/latest.json" \
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call passes '{}' into make_json, which results in "benchmarks": {{}} (invalid JSON). For an empty baseline, invoke make_json "$TMPDIR/baseline.json" with no benchmark args (or update make_json to avoid double-wrapping).

Copilot uses AI. Check for mistakes.
Comment on lines +100 to +102
@test "absolute threshold passes when under limit" {
make_json "$TMPDIR/baseline.json" '{}'
make_json "$TMPDIR/latest.json" \
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call passes '{}' into make_json, which results in "benchmarks": {{}} (invalid JSON). Use make_json "$TMPDIR/baseline.json" with no additional args (or make make_json accept a raw JSON object) to produce an empty benchmarks map.

Copilot uses AI. Check for mistakes.
Comment on lines +28 to +32
- uses: actions/checkout@v6
- name: Install bats
run: sudo apt-get install -y bats
- name: Run tests
run: bats tests/
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bats tests invoke scripts/compare.sh and scripts/format-release.sh, both of which hard-require jq and exit non-zero if it’s missing. The test job should install jq explicitly (or add a dedicated setup step) to keep CI deterministic across runner image changes.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: add tests for benchmark shell scripts

2 participants