bench(resolver): repurpose criterion at the rez_solver hot path#77
Merged
Conversation
The previous `cargo bench` lived in `rer-version` and measured the legacy `Requirement::from(&str)` parser plus `RerVersion::try_from` on a handful of strings. The legacy `Requirement` type isn't on the solver hot path (`rez_solver` has its own faithful `Requirement`), so seven of the eleven benchmarked cases measured dead-on-the-hot-path code; nothing exercised the `PackageVariantSlice` / `Requirement:: conflicts_with` / `VersionRange::union` work that the recent perf stack actually optimised. Replace it with a focused micro-benchmark suite under `crates/rer-resolver/benches/solver_micro.rs`: - **`VersionRange`**: `union`, `intersection`, `intersects` across disjoint / overlap / subset / any-vs-narrow shapes. - **`Requirement`**: `parse(unique)` (fresh string every call), `parse(memoised)` (thread-local cache hit), and three `conflicts_with` shapes (compatible, conflict, other-family). - **`PackageVariantSlice`**: `intersect` (any → Unchanged, narrow → Narrowed, drop-all → Empty), `reduce_by` (fast-path with unrelated family — the 98 % case — full body without and with reductions), and `extract` (common-dep extraction plus the O(1) exhausted → None early-return added in #71). - **`Solver`**: small full solves of representative shapes (single, pair, triple-with-pin, app). The bench uses a fixed in-memory synthetic repo (six families: python, qt, maya, nuke, usd, app — versions and shared deps modelled on the real shapes from the rez benchmark) so it's deterministic, fully reproducible from any checkout, and doesn't depend on `scripts/prepare_benchmark_data.py`. Drop `criterion` and the old `[[bench]]` block from `rer-version`'s manifest and delete `crates/rer-version/benches/main.rs`. Add `criterion` + the new `[[bench]]` block to `rer-resolver`. The macro 188-case benchmark (`examples/rez_benchmark_dataset`) remains the canonical end-to-end perf signal; the micro-bench complements it with stable, sub-microsecond unit-level numbers for regression baselines. ## Sample numbers (smoke run, this machine) ``` Requirement/conflicts_with(other-family) 3 ns ← early-out path Requirement/conflicts_with(conflict) 71 ns Requirement/conflicts_with(compatible) 137 ns PackageVariantSlice/intersect(any) 24 ns PackageVariantSlice/intersect(narrow) 145 ns PackageVariantSlice/reduce_by(fast) 30 ns ← 98 % path in the solver PackageVariantSlice/reduce_by(full) 2.2 µs PackageVariantSlice/extract(exhausted) 20 ns ← length-compare from #71 PackageVariantSlice/extract(common dep) 4.4 µs Solver/single 15 µs Solver/pair 33 µs Solver/app 92 µs ``` ## Suggested workflow ```text # Capture a baseline on main: cargo bench -p rer-resolver --bench solver_micro -- --save-baseline main # Compare a perf branch against it: cargo bench -p rer-resolver --bench solver_micro -- --baseline main ``` Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The previous `cargo bench` lived in `rer-version` and measured the legacy `Requirement::from(&str)` parser plus `RerVersion::try_from` on a handful of strings. The legacy `Requirement` type isn't on the solver hot path (`rez_solver` has its own faithful `Requirement`), so 7 of the 11 benched cases measured dead-on-the-hot-path code; nothing exercised the `PackageVariantSlice` / `Requirement::conflicts_with` / `VersionRange::union` work that the recent perf stack actually optimised.
Replace it with a focused micro-benchmark suite under `crates/rer-resolver/benches/solver_micro.rs`.
What's benched
extractable()via length compare #71).The bench uses a fixed in-memory synthetic repo (six families: python, qt, maya, nuke, usd, app — versions and shared deps modelled on the real shapes from the rez benchmark). Deterministic, reproducible from any checkout, no `scripts/prepare_benchmark_data.py` dependency.
The macro 188-case benchmark (`examples/rez_benchmark_dataset`) remains the canonical end-to-end perf signal. This micro-bench complements it with stable, sub-microsecond unit-level numbers for regression baselines.
Smoke-run numbers (this machine, `--quick`)
```
Requirement/conflicts_with(other-family) 3 ns ← early-out path
Requirement/conflicts_with(conflict) 71 ns
Requirement/conflicts_with(compatible) 137 ns
PackageVariantSlice/intersect(any) 24 ns
PackageVariantSlice/intersect(narrow) 145 ns
PackageVariantSlice/reduce_by(fast) 30 ns ← 98 % path in the solver
PackageVariantSlice/reduce_by(full) 2.2 µs
PackageVariantSlice/extract(exhausted) 20 ns ← length-compare from #71
PackageVariantSlice/extract(common dep) 4.4 µs
Solver/single 15 µs
Solver/pair 33 µs
Solver/app 92 µs
```
The `reduce_by(fast)` and `extract(exhausted)` numbers are the visible payoff of #68 and #71 respectively — making sure future refactors don't regress those is the whole point of having this bench.
Suggested workflow
```text
Capture a baseline on main:
cargo bench -p rer-resolver --bench solver_micro -- --save-baseline main
Compare a perf branch against it:
cargo bench -p rer-resolver --bench solver_micro -- --baseline main
```
Other changes
Verification
🤖 Generated with Claude Code