Skip to content

bench(resolver): repurpose criterion at the rez_solver hot path#77

Merged
doubleailes merged 1 commit into
mainfrom
criterion-solver-bench
May 15, 2026
Merged

bench(resolver): repurpose criterion at the rez_solver hot path#77
doubleailes merged 1 commit into
mainfrom
criterion-solver-bench

Conversation

@doubleailes
Copy link
Copy Markdown
Owner

Summary

The previous `cargo bench` lived in `rer-version` and measured the legacy `Requirement::from(&str)` parser plus `RerVersion::try_from` on a handful of strings. The legacy `Requirement` type isn't on the solver hot path (`rez_solver` has its own faithful `Requirement`), so 7 of the 11 benched cases measured dead-on-the-hot-path code; nothing exercised the `PackageVariantSlice` / `Requirement::conflicts_with` / `VersionRange::union` work that the recent perf stack actually optimised.

Replace it with a focused micro-benchmark suite under `crates/rer-resolver/benches/solver_micro.rs`.

What's benched

  • `VersionRange` — `union`, `intersection`, `intersects` across disjoint / overlap / subset / any-vs-narrow shapes.
  • `Requirement` — `parse(unique)` (fresh string every call), `parse(memoised)` (thread-local cache hit), and three `conflicts_with` shapes (compatible, conflict, other-family).
  • `PackageVariantSlice` — `intersect` (any → Unchanged, narrow → Narrowed, drop-all → Empty), `reduce_by` (fast-path with unrelated family — the 98 % case — full body without and with reductions), and `extract` (common-dep extraction plus the O(1) exhausted → None early-return from perf(resolver): O(1) extractable() via length compare #71).
  • `Solver` — small full solves of representative shapes (single, pair, triple-with-pin, app).

The bench uses a fixed in-memory synthetic repo (six families: python, qt, maya, nuke, usd, app — versions and shared deps modelled on the real shapes from the rez benchmark). Deterministic, reproducible from any checkout, no `scripts/prepare_benchmark_data.py` dependency.

The macro 188-case benchmark (`examples/rez_benchmark_dataset`) remains the canonical end-to-end perf signal. This micro-bench complements it with stable, sub-microsecond unit-level numbers for regression baselines.

Smoke-run numbers (this machine, `--quick`)

```
Requirement/conflicts_with(other-family) 3 ns ← early-out path
Requirement/conflicts_with(conflict) 71 ns
Requirement/conflicts_with(compatible) 137 ns
PackageVariantSlice/intersect(any) 24 ns
PackageVariantSlice/intersect(narrow) 145 ns
PackageVariantSlice/reduce_by(fast) 30 ns ← 98 % path in the solver
PackageVariantSlice/reduce_by(full) 2.2 µs
PackageVariantSlice/extract(exhausted) 20 ns ← length-compare from #71
PackageVariantSlice/extract(common dep) 4.4 µs
Solver/single 15 µs
Solver/pair 33 µs
Solver/app 92 µs
```

The `reduce_by(fast)` and `extract(exhausted)` numbers are the visible payoff of #68 and #71 respectively — making sure future refactors don't regress those is the whole point of having this bench.

Suggested workflow

```text

Capture a baseline on main:

cargo bench -p rer-resolver --bench solver_micro -- --save-baseline main

Compare a perf branch against it:

cargo bench -p rer-resolver --bench solver_micro -- --baseline main
```

Other changes

  • Drop `criterion` + the old `[[bench]]` block from `rer-version`'s manifest.
  • Delete `crates/rer-version/benches/main.rs`.
  • Update CLAUDE.md's command reference (`cargo bench` description).

Verification

  • `cargo build` — clean.
  • `cargo test` — passes.
  • `cargo bench -p rer-resolver --bench solver_micro -- --quick` — every target runs cleanly.

🤖 Generated with Claude Code

The previous `cargo bench` lived in `rer-version` and measured the
legacy `Requirement::from(&str)` parser plus `RerVersion::try_from` on
a handful of strings. The legacy `Requirement` type isn't on the
solver hot path (`rez_solver` has its own faithful `Requirement`), so
seven of the eleven benchmarked cases measured dead-on-the-hot-path
code; nothing exercised the `PackageVariantSlice` / `Requirement::
conflicts_with` / `VersionRange::union` work that the recent perf
stack actually optimised.

Replace it with a focused micro-benchmark suite under
`crates/rer-resolver/benches/solver_micro.rs`:

- **`VersionRange`**: `union`, `intersection`, `intersects` across
  disjoint / overlap / subset / any-vs-narrow shapes.
- **`Requirement`**: `parse(unique)` (fresh string every call),
  `parse(memoised)` (thread-local cache hit), and three
  `conflicts_with` shapes (compatible, conflict, other-family).
- **`PackageVariantSlice`**: `intersect` (any → Unchanged, narrow →
  Narrowed, drop-all → Empty), `reduce_by` (fast-path with unrelated
  family — the 98 % case — full body without and with reductions),
  and `extract` (common-dep extraction plus the O(1) exhausted →
  None early-return added in #71).
- **`Solver`**: small full solves of representative shapes
  (single, pair, triple-with-pin, app).

The bench uses a fixed in-memory synthetic repo (six families: python,
qt, maya, nuke, usd, app — versions and shared deps modelled on the
real shapes from the rez benchmark) so it's deterministic, fully
reproducible from any checkout, and doesn't depend on
`scripts/prepare_benchmark_data.py`.

Drop `criterion` and the old `[[bench]]` block from `rer-version`'s
manifest and delete `crates/rer-version/benches/main.rs`. Add
`criterion` + the new `[[bench]]` block to `rer-resolver`.

The macro 188-case benchmark (`examples/rez_benchmark_dataset`)
remains the canonical end-to-end perf signal; the micro-bench
complements it with stable, sub-microsecond unit-level numbers for
regression baselines.

## Sample numbers (smoke run, this machine)

```
Requirement/conflicts_with(other-family)   3   ns   ← early-out path
Requirement/conflicts_with(conflict)      71   ns
Requirement/conflicts_with(compatible)   137   ns
PackageVariantSlice/intersect(any)        24   ns
PackageVariantSlice/intersect(narrow)    145   ns
PackageVariantSlice/reduce_by(fast)       30   ns   ← 98 % path in the solver
PackageVariantSlice/reduce_by(full)      2.2  µs
PackageVariantSlice/extract(exhausted)    20   ns   ← length-compare from #71
PackageVariantSlice/extract(common dep)  4.4  µs
Solver/single                             15   µs
Solver/pair                               33   µs
Solver/app                                92   µs
```

## Suggested workflow

```text
# Capture a baseline on main:
cargo bench -p rer-resolver --bench solver_micro -- --save-baseline main

# Compare a perf branch against it:
cargo bench -p rer-resolver --bench solver_micro -- --baseline main
```

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@qodo-code-review
Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@doubleailes doubleailes merged commit 1d61a24 into main May 15, 2026
24 checks passed
@doubleailes doubleailes deleted the criterion-solver-bench branch May 15, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant