Skip to content

chore(bench): skip multi-threaded cases in CodSpeed memory mode#244

Merged
hardfist merged 3 commits into
mainfrom
chore/rename-multi-threaded-bench
May 26, 2026
Merged

chore(bench): skip multi-threaded cases in CodSpeed memory mode#244
hardfist merged 3 commits into
mainfrom
chore/rename-multi-threaded-bench

Conversation

@stormslowly
Copy link
Copy Markdown
Collaborator

@stormslowly stormslowly commented May 26, 2026

Why

CodSpeed memory mode runs Valgrind massif, which conflates allocations across worker threads and produces noisy, non-actionable deltas for the multi-threaded resolver benches. Skip them in that mode so memory deltas only come from the single-threaded cases.

What

  1. Rename the two threaded bench cases to share a [multi-threaded] prefix (matches the existing [single-threaded]resolve with many extensions style):
    • multi-thread[multi-threaded]resolve
    • resolve from symlinks multi thread[multi-threaded]resolve from symlinks
  2. Gate them on the CODSPEED_RUNNER_MODE env var (set by the CodSpeed action). They keep running in the simulation step and are skipped only in memory mode.

No workflow change needed — the existing mode: memory action already exports CODSPEED_RUNNER_MODE=memory into the bench process.

Rename the two threaded resolver bench cases so they share a common
`[multi-threaded]` prefix, allowing a single regex (e.g.
`--skip '\[multi-threaded\]'`) to select or exclude them at runtime.
Copilot AI review requested due to automatic review settings May 26, 2026 09:54
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR standardizes the naming of the multi-threaded Criterion benchmark cases in benches/resolver.rs by adding a shared [multi-threaded] prefix, enabling easy selection/skipping via regex (e.g., when running CodSpeed in modes where multi-threaded throughput isn’t meaningful).

Changes:

  • Renamed the "multi-thread" benchmark case to "[multi-threaded]resolve".
  • Renamed "resolve from symlinks multi thread" to "[multi-threaded]resolve from symlinks".

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 26, 2026

Merging this PR will improve performance by 3.2%

⚡ 1 improved benchmark
✅ 87 untouched benchmarks
🆕 4 new benchmarks
⏩ 4 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 Memory resolver[[multi-threaded]resolve from symlinks] N/A 11.6 MB N/A
🆕 Memory resolver[[multi-threaded]resolve] N/A 10.9 MB N/A
🆕 Simulation resolver[[multi-threaded]resolve] N/A 59 ms N/A
🆕 Simulation resolver[[multi-threaded]resolve from symlinks] N/A 76.5 ms N/A
Simulation specifier/branches[multi-question] 940 ns 910.8 ns +3.2%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing chore/rename-multi-threaded-bench (deeba50) with main (4bdda7b)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

CodSpeed memory mode runs Valgrind massif, which conflates allocations
across worker threads and cannot produce a meaningful signal for the
multi-threaded resolver benches. Gate them on `CODSPEED_RUNNER_MODE`
(set by the CodSpeed action) so they only run in the simulation step.
@stormslowly stormslowly changed the title chore(bench): unify multi-threaded bench case names with [multi-threaded] prefix chore(bench): skip multi-threaded cases in CodSpeed memory mode May 26, 2026
@hardfist hardfist merged commit 9dd63ca into main May 26, 2026
@hardfist hardfist deleted the chore/rename-multi-threaded-bench branch May 26, 2026 13:07
@stormslowly stormslowly mentioned this pull request May 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants