Skip to content

feat(bench): publish release benchmark baselines#144

Merged
acgetchell merged 1 commit into
mainfrom
feat/138-release-benchmark-baselines
Jun 4, 2026
Merged

feat(bench): publish release benchmark baselines#144
acgetchell merged 1 commit into
mainfrom
feat/138-release-benchmark-baselines

Conversation

@acgetchell
Copy link
Copy Markdown
Owner

@acgetchell acgetchell commented Jun 4, 2026

  • Add a release-only benchmark workflow that saves full Criterion baselines for published releases and attaches the archived baseline to the GitHub Release.
  • Keep the regular benchmark workflow focused on PR and main-branch comparison runs.
  • Document how to restore archived release baselines for future performance comparisons.

Closes #138

Summary by CodeRabbit

  • Documentation

    • Improved benchmarking documentation with guidance on baseline persistence and restoration procedures.
    • Enhanced release documentation describing baseline handling during the release process.
  • Chores

    • Updated workflow documentation comments.

- Add a release-only benchmark workflow that saves full Criterion baselines for published releases and attaches the archived baseline to the GitHub Release.
- Keep the regular benchmark workflow focused on PR and main-branch comparison runs.
- Document how to restore archived release baselines for future performance comparisons.

Closes #138
@acgetchell acgetchell self-assigned this Jun 4, 2026
@acgetchell acgetchell enabled auto-merge June 4, 2026 22:52
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Jun 4, 2026

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3123a94e-d7fc-4e42-8c62-5af8dfe2123a

📥 Commits

Reviewing files that changed from the base of the PR and between ad660ba and 9497ca5.

📒 Files selected for processing (4)
  • .github/workflows/benchmarks.yml
  • .github/workflows/release-benchmarks.yml
  • docs/BENCHMARKING.md
  • docs/RELEASING.md

📝 Walkthrough

Walkthrough

This PR introduces release baseline capture infrastructure and documentation. A new GitHub Actions workflow runs on release publication, benchmarks both the vs_linalg and exact suites, and uploads the Criterion baseline archive to the GitHub Release. Documentation explains how baselines persist, how to restore them from releases, and how the workflow integrates into the release process.

Changes

Release Baseline Infrastructure

Layer / File(s) Summary
Release baseline capture workflow
.github/workflows/release-benchmarks.yml
New workflow triggered on release.published events that checks out the release tag, runs cargo bench for both vs_linalg and exact suites with Criterion baseline saving, packages the criterion directory into a versioned tar.gz archive, uploads it as a temporary artifact, attaches it to the GitHub release via gh release upload --clobber, and appends the asset filename to GITHUB_STEP_SUMMARY.
Baseline persistence and restoration guide
docs/BENCHMARKING.md
Clarified that git checkout persists Criterion baselines, cargo clean removes them, and GitHub Releases provide compressed archives for later restoration. Extended "Latest vs last" workflow with two conditional paths: direct comparison if baseline exists locally under target/criterion/, or a restore flow (download release asset, untar into target/criterion/) followed by rerunning measurements and comparison when baseline is missing.
Release process integration
docs/RELEASING.md, .github/workflows/benchmarks.yml
Updated RELEASING.md to explain that release-saved baselines enable future comparisons and documented the Release Benchmarks workflow behavior (generating and attaching the durable la-stack-$TAG-criterion-baseline.tar.gz archive plus a short-lived debugging artifact). Reworded the top-level comment in benchmarks.yml from a redundant phrasing to the concise "Detect performance regressions."

Sequence Diagram

sequenceDiagram
  participant Release as GitHub Release
  participant Actions as GitHub Actions
  participant Cargo as Cargo Bench
  participant Artifact as Artifact Upload
  participant GHAsset as GH Release Asset
  participant Summary as GITHUB_STEP_SUMMARY
  
  Release->>Actions: published event
  Actions->>Actions: checkout release tag
  Actions->>Cargo: cargo bench (vs_linalg)
  Cargo->>Cargo: save Criterion baseline
  Actions->>Cargo: cargo bench (exact with features)
  Cargo->>Cargo: save Criterion baseline
  Actions->>Actions: tar.gz criterion/ directory
  Actions->>Artifact: upload temporary artifact (30-day)
  Actions->>GHAsset: gh release upload --clobber
  GHAsset->>Release: attach la-stack-$TAG-criterion-baseline.tar.gz
  Actions->>Summary: write asset filename and path
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

  • acgetchell/la-stack#143: Implements the scripts/bench_compare.py baseline comparison logic and test infrastructure that this PR's new release-benchmarks workflow and baseline restoration documentation depend on for comparing current code against saved release baselines.

Poem

🐰 Baselines captured on release day,
Benchmarks saved in archives to stay,
Future comparisons, swift and clear,
Regressions spotted from branch to branch, year after year! 🏃

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main feature: publishing release benchmark baselines as part of the release workflow.
Linked Issues check ✅ Passed The PR implements all key coding requirements from issue #138: new release benchmark workflow, baseline archiving, documentation updates, and restoration instructions for performance comparisons.
Out of Scope Changes check ✅ Passed All changes are directly aligned with issue #138 objectives: workflow updates, documentation of baseline persistence and restoration, and release asset attachment.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/138-release-benchmark-baselines

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 4, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.61%. Comparing base (ad660ba) to head (9497ca5).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #144   +/-   ##
=======================================
  Coverage   99.61%   99.61%           
=======================================
  Files           5        5           
  Lines        2846     2846           
=======================================
  Hits         2835     2835           
  Misses         11       11           
Flag Coverage Δ
unittests 99.61% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@acgetchell acgetchell merged commit 5bde071 into main Jun 4, 2026
17 checks passed
@acgetchell acgetchell deleted the feat/138-release-benchmark-baselines branch June 4, 2026 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add first-class cross-release performance comparison tooling

1 participant