From 53f1707bba062bda5c2a222b808ea3eb5a50e724 Mon Sep 17 00:00:00 2001 From: Philippe Llerena Date: Fri, 15 May 2026 20:49:11 +0200 Subject: [PATCH] docs: add "Wiring pyrer into rez" integration guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `pyrer` is the solver hotpath only — rez still owns discovery, env build, the context lifecycle. The README has the `pyrer.solve` API but doesn't say how to actually slot it behind a normal `rez env` flow. Filling that gap. New Zola page under getting-started: - The integration model (what pyrer is / isn't), with an ASCII pipeline diagram showing where rez stays and where pyrer takes over. - `build_pyrer_repo(package_paths)` — walks `iter_package_families` and produces the JSON shape `pyrer.solve` expects. - Result translation back to rez `Variant` objects via `rez.packages.get_package(name, version).get_variant(idx)`. - A minimum-viable monkey-patch of `rez.resolver.Resolver._solve` that delegates to pyrer on the happy path and falls back to rez's own solver for any non-default config (custom orderer, filters, late-binding requires). - Caveats section explicitly listing what pyrer does *not* model yet: `VariantSelectMode::intersection_priority` (issue #63), `@early` / `@late` requires, custom `PackageOrder` / `PackageFilter`, variant-index parity in the differential. - A sanity-check loop for diffing pyrer vs rez on your own repo. - A note on running many resolves against the same repo: build the dict once, plus pointer to `pyrer`'s shared variant cache. Also: - Link the new page from the introduction's "Next steps" list so it's discoverable from the docs front page. - Add a compressed reference block in the README's "Using it from Python" section (the same three steps: walk → solve → translate), with a pointer to the full docs page. zola build / zola check both clean; 16 pages (was 15). Co-Authored-By: Claude Opus 4.7 --- README.md | 42 +++ .../docs/getting-started/introduction.md | 3 + .../docs/getting-started/rez-integration.md | 262 ++++++++++++++++++ 3 files changed, 307 insertions(+) create mode 100644 docs/content/docs/getting-started/rez-integration.md diff --git a/README.md b/README.md index cf92ccf..49907fc 100644 --- a/README.md +++ b/README.md @@ -97,6 +97,48 @@ print(result.resolved) # [("app", "1.0.0", None), ("lib", "2.0.0", None)] `solve()` reports failures and bad input via `result.status` (`"solved"` / `"failed"` / `"error"`), never as a Python exception. +### Wiring `pyrer` behind `rez` + +`pyrer` only does the solve; `rez` still does package discovery, env +construction, and the whole `ResolvedContext` lifecycle. To plug +`pyrer` in behind a normal `rez env` / `ResolvedContext` flow: + +1. Walk rez's package paths into the `pyrer` repo shape — once per + process, reusable across many solves: + + ```python + from rez.packages import iter_package_families + + def build_pyrer_repo(package_paths): + repo = {} + for fam in iter_package_families(paths=package_paths): + repo[fam.name] = { + str(pkg.version): { + "requires": [str(r) for r in (pkg.requires or [])], + "variants": [ + [str(r) for r in v] for v in (pkg.variants or []) + ], + } + for pkg in fam.iter_packages() + } + return repo + ``` + +2. Call `pyrer.solve(requests, json.dumps(repo))` instead of + `rez.solver.Solver.solve()`. + +3. Translate `result.resolved` back into rez `Variant` objects with + `rez.packages.get_package(name, version).get_variant(idx)` and feed + them into whatever you'd normally hand to `ResolvedContext`. + +The +[Wiring `pyrer` into `rez`](https://doubleailes.github.io/rer/docs/getting-started/rez-integration/) +guide has the full walkthrough — a minimal monkey-patch of +`Resolver._solve`, a fallback for configs `pyrer` doesn't model yet +(`intersection_priority`, `@early` / `@late` requires, custom +orderers / filters), and a sanity-check loop for diffing `pyrer` +against rez on your own repo. + ## Building & testing ```bash diff --git a/docs/content/docs/getting-started/introduction.md b/docs/content/docs/getting-started/introduction.md index 5cd0554..ee30055 100644 --- a/docs/content/docs/getting-started/introduction.md +++ b/docs/content/docs/getting-started/introduction.md @@ -69,6 +69,9 @@ builds the environment, and manages contexts. rer just does the solving. - **[Quick Start →](../quick-start/)** — install rer and run your first resolve from Python or Rust. +- **[Wiring `pyrer` into `rez` →](../rez-integration/)** — how to plug + `pyrer` in behind a normal `rez env` / `ResolvedContext` flow, with a + minimal shim and the caveats. - **[Engineering notes →](../../engineering/)** — design decisions and measurements behind the port (e.g. why some rez optimisations are intentionally absent). diff --git a/docs/content/docs/getting-started/rez-integration.md b/docs/content/docs/getting-started/rez-integration.md new file mode 100644 index 0000000..a45f2cf --- /dev/null +++ b/docs/content/docs/getting-started/rez-integration.md @@ -0,0 +1,262 @@ ++++ +title = "Wiring `pyrer` into `rez`" +description = "How to use pyrer as the solver backend behind a normal rez workflow." +date = 2026-05-15T08:30:00+00:00 +updated = 2026-05-15T08:30:00+00:00 +draft = false +weight = 30 +sort_by = "weight" +template = "docs/page.html" + +[extra] +lead = "How to plug pyrer into a normal rez workflow: rez still handles package discovery and environment construction; pyrer just does the solve." +toc = true +top = false ++++ + +## What `pyrer` is, and what it is not + +`pyrer` is **only the solver hotpath** — the rez-faithful phase-based +backtracking algorithm, ported to Rust and called from Python through +PyO3. It is *not* a replacement for `rez`. It does not: + +- discover packages on the filesystem, +- parse `package.py` (it takes pre-parsed requirements as strings), +- build the runtime environment (PATH, env vars, shell hooks), +- handle the `rxt` context lifecycle, suites, or context bundling. + +`rez` keeps doing all of that. `pyrer` is dropped in at the one step +where the cost lives: solving the version constraints. + +The minimum integration looks like: + +``` +┌────────────────────────────────────────────┐ +│ rez: iter_package_families / iter_packages │ ← package discovery +└────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────┐ +│ build a pyrer repo dict (name → version │ +│ → {requires, variants}) │ ← one-time conversion +└────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────┐ +│ pyrer.solve(requests, json.dumps(repo)) │ ← the fast bit +└────────────────────────────────────────────┘ + │ + ▼ +┌────────────────────────────────────────────┐ +│ resolve → rez Variant objects → │ +│ ResolvedContext / env build │ ← rez again +└────────────────────────────────────────────┘ +``` + +## Building the `pyrer` repo from `rez` + +`pyrer.solve()` expects an in-memory repository in the schema +`{name: {version: {"requires": [...], "variants": [[...]]}}}` — the +same shape rer's tests use. + +A minimal converter from a rez package path looks like this: + +```python +from rez.packages import iter_package_families + + +def build_pyrer_repo(package_paths): + """Walk rez's package paths and produce the JSON-shaped repo pyrer wants.""" + repo = {} + for family in iter_package_families(paths=package_paths): + versions = {} + for pkg in family.iter_packages(): + versions[str(pkg.version)] = { + "requires": [str(r) for r in (pkg.requires or [])], + "variants": [ + [str(r) for r in variant] + for variant in (pkg.variants or []) + ], + } + repo[family.name] = versions + return repo +``` + +Two notes on this step: + +- It is **eager** — every package on every path is loaded. `rez` + normally loads lazily; the trade-off is one upfront cost vs many + small ones during the solve. On a real repo, eager loading is + typically a few seconds; on the rez 188-case benchmark it is the + dominant pre-solve cost. +- If you're running many resolves against the same repo (CI, batch + validation, a long-lived daemon), build the dict **once** and reuse + it. Better still, share a `pyrer.make_shared_cache()` between + `Solver::new_with_cache(...)` calls so even rer's internal variant + cache lives across solves. See the + [`SharedVariantCache`](https://github.com/doubleailes/rer/blob/main/crates/rer-resolver/src/rez_solver/context.rs) + type. + +## Solving + +```python +import json +import pyrer + +repo = build_pyrer_repo(["/sw/pkg", "/sw/site"]) +repo_json = json.dumps(repo) + +result = pyrer.solve(["maya-2024", "nuke-14"], repo_json) + +print(result.status) # "solved" | "failed" | "error" +print(result.solve_time_ms) # wall-clock of just the Rust solve +for name, version, variant_idx in result.resolved: + print(name, version, variant_idx) +``` + +`status` distinguishes: + +- **`"solved"`** — `result.resolved` is the resolution as a list of + `(name, version, variant_index)` tuples. `variant_index` is `None` + for packages with no `variants` defined. +- **`"failed"`** — a real resolve conflict; `result.failure_description` + has a human-readable reason. +- **`"error"`** — bad input (malformed repo JSON, unparseable + requirement string, missing top-level package). + +No Python exception is ever raised from `pyrer.solve()` — even a +panic inside the Rust solver is caught and reported as `"error"`. + +## Translating the result back to `rez` + +To hand the resolution back to rez (for environment construction, +context bundling, etc.), look the resolved tuples up against rez's own +package iterator: + +```python +from rez.packages import get_package, get_variant + + +def resolve_to_rez_variants(result, package_paths): + """Turn pyrer's (name, version, variant_index) tuples into rez Variants.""" + variants = [] + for name, version, idx in result.resolved: + pkg = get_package(name, version, paths=package_paths) + if pkg is None: + raise RuntimeError(f"package vanished after solve: {name}-{version}") + # idx is None for packages with no variants — rez models that + # as a single variant with index 0 internally. + variants.append(pkg.get_variant(idx if idx is not None else 0)) + return variants +``` + +These `Variant` objects can be fed into rez's normal context machinery +(see `rez.resolved_context.ResolvedContext` — you'll want to look at +how its internal solver result is normally consumed and substitute the +list above). For most workflows the most useful thing is to call +`rez.rex.bind` / `Variant.apply_value` on each variant against an +`ActionInterpreter`, which is the same code rez runs after its own +solve. + +## A complete monkey-patch shim + +If you want pyrer to transparently accelerate `rez env` / +`ResolvedContext` without changing call sites, the smallest sound +patch is to replace `rez.solver.Solver.solve` with a delegating +implementation. This is non-trivial to get right (rez's `Solver` +exposes a rich status surface — `phase_stack`, `failure_reason`, +graph rendering, callback support) so the patch is best kept narrow: +intercept the happy path, fall back to the real rez solver on any +non-default config (custom orderer, `late` binding requires, +`@early` evaluation, etc.). + +The minimum viable shim — for studios with default-configured repos — +looks roughly like: + +```python +import json +import pyrer +import rez.solver as _rez_solver +import rez.resolver as _rez_resolver + +_original_resolve = _rez_resolver.Resolver._solve + + +def _pyrer_resolve(self): + # Fall back to rez on anything pyrer doesn't support yet. + if self.package_filter or self.package_orderers: + return _original_resolve(self) + + repo = build_pyrer_repo(self.package_paths) + requests = [str(r) for r in self.package_requests] + result = pyrer.solve(requests, json.dumps(repo)) + + if result.status != "solved": + return _original_resolve(self) # let rez produce the canonical failure + + self.resolved_packages_ = resolve_to_rez_variants( + result, self.package_paths, + ) + self.status_ = _rez_solver.SolverStatus.solved + return self + + +_rez_resolver.Resolver._solve = _pyrer_resolve +``` + +Load this once at process start (e.g. via a `rezconfig.py`'s +`plugin_path` entry or a `sitecustomize.py`) and any `rez env`, +`rez build`, `rez-bundle` etc. running in that process will route +through `pyrer` for the solve. + +## Caveats and what isn't supported yet + +`pyrer.solve` is the solver only. The following are **not** modelled +by it — if your studio depends on any of these, fall back to rez's +solver for those resolves: + +- **`VariantSelectMode::intersection_priority`.** `pyrer` implements + rez's default `version_priority` only; see + [issue #63](https://github.com/doubleailes/rer/issues/63). +- **`@early` / `@late` binding requires.** `pyrer` takes already- + parsed strings; if a package's requires depend on the resolve + context, rez has to evaluate them first. +- **Custom package orderers and filters.** Anything that hooks into + `PackageOrder` / `PackageFilter` runs in rez; the integration shim + above falls back when these are configured. +- **Cyclic-failure detail.** Both solvers detect cycles; the human- + readable failure message differs in wording. +- **Variant-index parity.** The differential test currently checks + the resolved `(name, version)` set, not the variant index — variant + selection is rez-faithful by construction but is not enforced by + the test suite. See the [README's *Validated 1:1* note](../../../../#status). + +## Sanity-checking against rez + +To make sure `pyrer` agrees with `rez`'s solver on your own repo, +generate a small set of representative requests and diff the +resolutions: + +```python +from rez.resolved_context import ResolvedContext + +for request in your_real_requests: + rer_result = pyrer.solve(request, repo_json) + rez_ctx = ResolvedContext(request, package_paths=["/sw/pkg"]) + rer_set = {(n, v) for n, v, _ in rer_result.resolved} + rez_set = {(v.name, str(v.version)) for v in rez_ctx.resolved_packages} + assert rer_set == rez_set, f"diverge on {request}" +``` + +If any case diverges, open an issue with the request and a minimal +package set that reproduces — the project's correctness bar is "match +rez 1:1" and divergence is a release blocker. + +## See also + +- [Quick Start →](../quick-start/) — the basic `pyrer.solve` API in + isolation, without any rez integration. +- [Engineering notes →](../../engineering/) — design decisions behind + the port (e.g. why some rez optimisations are intentionally absent). +- [rez integration in the repo README](https://github.com/doubleailes/rer/blob/main/README.md) + — short reference card.