pgw#1507: the varena facade — serve a real pipeline out of the arena behind the pgw#1497 planner contract - #1068
Merged
Merged
Conversation
PaulFidika
force-pushed
the
1507-varena-facade
branch
from
August 19, 2026 23:39
85f9749 to
e7628ca
Compare
PaulFidika
marked this pull request as ready for review
August 19, 2026 23:40
…as the mechanism
pgw#1497 shipped the CONTRACT (plan_residency, the {VRAM, RAM} pair,
engage/rebudget/partial_unload/demote_to_host) with ONE mechanism behind it: a
pinned-CPU tail cast per forward into a shared ring buffer. This is the SECOND
mechanism behind the SAME contract.
`models/arena_residency.py`:
* **One reservation, stable addresses.** One varena Arena per model holds the
lease's VRAM half; one Reservation holds a virtual range for every managed
weight and that range never moves. Resident leaves are `back()`-ed and the
module's parameter is a zero-copy DLPack view; streamed leaves are unbacked
and cost zero VRAM. Promote/demote/rebudget move PHYSICAL PAGES under a fixed
pointer — which is the property phase 3 needs to lift compiled⊥offload, and
which is NOT lifted here.
* **The granularity tax is charged to the planner, not discovered later.** CUDA
VMM maps in 2 MiB chunks, so independent per-leaf back/unback requires
per-leaf granularity-aligned regions — two leaves sharing a chunk could never
release it, and would race if one were in flight. The layout therefore prices
each region at its ALIGNED SPAN, so `plan.resident_bytes` equals the bytes the
arena really maps and `plan.fits` is exact instead of optimistic in the
direction that OOMs. `ArenaLayout.tax_bytes` names the total; leaves that can
never stream are packed into ONE core region and pay one chunk remainder
between them instead of one each (226 sd1.5 unet leaves, 67 MB, one remainder).
* **UnbackRing — the correctness point.** A post-hook fires when the leaf's
kernels are ENQUEUED, not when they have run, so unbacking there unmaps memory
a live kernel is reading. The post-hook records a CUDA event and the region
joins a `depth`-deep ring; the unback happens when a later leaf needs the room,
after that event completed. `depth` IS the planner's `streams`, so the ring
holds exactly the in-flight window `plan_residency` already reserved.
* **Two refill sources, chosen by the RAM half, not by a flag.** Host mirror
present -> pinned H2D on fenced offload streams. No mirror (`host_mirror=False`,
i.e. a stated RAM half of zero) -> straight off disk through varena's
RefillEngine. That is pgw#1497's owed RAM-half enforcement arriving as a
DEGRADE rather than a refusal. A meta-initialised tree is COLD, not broken:
its weights go disk -> arena and never occupy a torch allocation at all.
* **`safetensors_triples` is the seam.** (path, offset, len) from the header's
own data_offsets, no tensor materialised. pgw#1498's CAS-native ingest arrives
as a different producer of the same dict — one constructor argument.
* LoRA semantics are identical to the streamed rung because they are the SAME
code: adapters are detected by `discover_leaves` and land in the always-resident
core, exactly as pgw#1497 forces them resident.
`models/stream_residency.py`: the leaf walk, the tensor helpers and the 512-byte
alignment are now public and SHARED (`discover_leaves`, `own_tensors`,
`bind_tensor`, `is_streamable_leaf`, `is_adapter_name`, `tensor_bytes`,
`aligned`). Two mechanisms behind one contract must agree leaf-for-leaf or their
plans are not comparable, and a second copy of that walk is how they would
silently stop agreeing. No behaviour change; pgw#1497's 20 tests still pass.
Deliberately absent, per varena's minimalism ruling applied to its caller: no
NVML, no probe, no prefetch policy, no eviction heuristic. The budget is the
lease's and the split is pgw#1497's planner. varena itself is UNMODIFIED (pinned
at master 9b287a4).
16 tests over REAL nn.Module trees and REAL safetensors files, every one
red-armed: unaligned spans -> 2 red; +1 on the triple offset -> 1 red; no core
packing -> 4 red. ruff + mypy clean.
GPU legs (bitwise identity, demote/promote cycles, the pricing rerun, cold load)
follow in the next commit — the card is arbitrated and this half is CPU-only.
…exists for is a LOSS on the streaming axis RTX 4070, sd1.5, 512^2, 25 steps, CFG 7.5, fp16, seed 1507, latent output compared with `torch.equal`. ONE config, warmup, best of 2, load 3.17 at start. Raw JSON in `benchmarks/pgw1507/`. ## Identity and cycles — clean Eager resident vs arena resident-equivalent: BITWISE IDENTICAL. Three demote/promote cycles BETWEEN requests: mapped 2122 -> 66 -> 2122 MiB each time, signature 1->239->239, 239->477->477, 477->715->715 (advances on demote, stable across promote — `back()` is signature-neutral, as varena#1 claims), every resident region confirmed backed by asking the DRIVER rather than our own bookkeeping, and the output after each re-promotion still bitwise identical. LoRA: 96 attached pairs -> 192 adapter leaves, ALL in the always-resident core, ZERO ever given a streamable region, while 36 of their base layers DO stream. Bitwise identical at full / 25% / 5%. Identical to the streamed rung because it is the same `discover_leaves`. ## The pricing table | arm | ms/step | x native | device peak | pgw#1497 software rung | |---|---|---|---|---| | eager resident | 107.6 | 1.00 | 2.68 GiB | 1.00 (119.6 ms) | | arena resident-equivalent | 110.0 | **1.02** | 3.10 GiB | - | | arena @50% | 302.3 | **2.81** | 1.94 GiB | **1.91** | | arena @25% | 425.5 | **3.95** | 1.49 GiB | **3.16** | | arena @5% | 454.5 | **4.22** | 1.24 GiB | **3.56** | The arena is ~25% SLOWER than the software rung at every streaming budget, and free (1.02x) when nothing streams. Stated plainly because it is the number the issue asked for. ## Why — measured, and the alternative explanation falsified first Micro-benchmark, same card: `back(4MiB)+unback(4MiB)` 97.8 us; `back()` on an already-backed range 0.2 us; DLPack export + `from_dlpack` 6.1 us; H2D 4 MiB pinned->arena 376.0 us vs pinned->ordinary torch buffer 368.5 us; `signature()` 0.1 us. So **arena memory is not slower for DMA** and the whole gap is ~98 us of page-table work per streamed leaf per forward: at 25%, ~464 page-ins/step x 98 us ~= 45 ms/step against a measured gap of 60 ms/step. The obvious competing explanation — the ring's CPU-side event block serialising the pipeline — was FALSIFIED before this one was believed: sweeping ring depth (streams 2/4/8/16) at the 25% budget gives 3.56x / 3.74x / 4.07x / 3.83x. Deeper is WORSE, because a deeper ring only reserves more in-flight window and evicts more resident leaves. Second cost, predicted CPU-side and confirmed to the byte: the granularity tax is 248 MiB = 13.2% (1.830 GiB of weights in 2.072 GiB of span, 239 regions). Charged to the planner, so `plan.resident_bytes` is exact — but the arena holds 13% less WEIGHT than the software rung at the same lease. Contract-level demote/promote, both mechanisms, same tree: arena 744 ms demote / 232 ms promote (2.008 GiB) vs software 630 / 157 ms (1.768 GiB) — per GiB 370/116 vs 356/89 ms, again the extra bytes plus the map cost. ## Cold load — the arm the arena WINS Byte-exact: 0 mismatched of 686 UNet tensors, page-cache warm AND evicted (`fadvise DONTNEED`; a "cold load" measured with the file in RAM measures memcpy). Warm: loader 0.31 s (5.49 GB/s) vs RefillEngine->arena 0.37 s. EVICTED: loader 3.89 s (0.44 GB/s) vs arena 3.03 s (0.57 GB/s) — **1.28x, because the meta-init tree never allocates a torch tensor for the weights at all.** ## Three defects the card found 1. **`pipeline.device` again.** The facade parked leaves on the host without stamping the handle pgw#1497's execution-device repair reads, so sd1.5 died in the first embedding with `index is on cpu`. Fixed where it belongs: `ArenaResidency.arm()` is the production entry point and discharges all three of `_apply_partial_stream`'s obligations — exclude AND place the unhookable components, filter the component vocabulary, stamp the handle and install the repair. The repair itself needed no change: it asks the handle where the model executes, so a second mechanism gets it by being stamped. 2. **`Parameter.data =` across the meta boundary.** torch REFUSES it ("incompatible tensor type") and the cold-load leg died there. A meta parameter has never held a byte and has no identity worth preserving, so it is REPLACED; every other bind still fills `.data` and preserves identity, which is what keeps hooks and LoRA wrappers working across promotes. 3. **239 submit/wait round trips.** Adopting a cold tree region by region drains the engine queue to empty each time, so io_uring's depth buys nothing — 0.83 s. `_adopt_cold` hands the engine every triple in ONE batch, both destinations at once: 0.37 s. That batched shape is also what pgw#1498's CAS ingest wants. 18 CPU tests (2 new: the LoRA clause and the meta-bind rule), 58 green with pgw#1497's. ruff + mypy clean. varena UNMODIFIED, pinned at master 9b287a4.
…med for its DOMAIN, and the unreached facade DECLARES its owner
Three CI findings, all correct, none worked around.
1. **mypy strict over tests.** `torch = pytest.importorskip("torch")` binds a
VARIABLE, so `torch.Tensor` in an annotation is not a type at all. Annotations
dropped rather than a stub added; `specs_for` got its return type.
2. **`lint_incident_test_names` (DESIGN-RULINGS 4.34b).** `test_arena_layout_pgw1507.py`
said WHEN it was written, not WHAT it exercises. Renamed to
`tests/test_arena_residency.py` — the domain — with the lineage kept as a
one-line comment where the ruling says to keep it. The baseline list only
shrinks, so adding a row was never an option and the guard is right that it
should not have been.
3. **`lint_unreached_surface`.** `ArenaResidency` and `safetensors_triples` have
no non-test call site, and the gate is TELLING THE TRUTH: nothing in `src/`
reaches the facade. That is a DECISION and it is now written down where the
guard demands — an `EXEMPT_TARGETS` row naming the owner of the wiring, which
the file's own contract calls "a DELETION THAT IS OWED, never a permanent
pardon". A ladder rung takes a MEASURED price and the measured price is worse
than `partial_stream`'s at every streaming budget, so putting it on the ladder
now would add a rung `select_auto_mode` must never choose. It gets a
production caller when roadmap phase 3 gives it the thing its stable virtual
addresses are for; if phase 3 is abandoned those two rows are the deletion
notice for the module.
PRE-EXISTING and NOT MINE, stated so nobody reads this PR's gate colour as its
own verdict: `fast gates` is red on `origin/master` and on every recent PR
(#1061, #1063, #1066, #1067 all show `fast gates: fail`). Reproduced locally on
this branch: `lint_unreached_surface` still reports six NEW unreached symbols in
`models/gguf_torch` — which is exactly what pgw#1498's own tracker entry says
about itself ("the module is correct but NOT REACHABLE") — and
`lint_incident_test_names` still reports pgw#1497's, #1498's and #1499's
issue-named modules. This branch adds no new violation of either.
PaulFidika
force-pushed
the
1507-varena-facade
branch
from
August 20, 2026 00:14
8defdd0 to
055782c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Roadmap phase 1 remainder. varena's primitives were merged and verified; nothing
served through them. This makes an arbitrary diffusers
nn.Moduletree hold itsweights in arena
Reservations, behind the SAME planner/profile contractmodels/stream_residency.pyshipped in pgw#1497 — so the rung swaps without anendpoint noticing.
varena pin:
master 9b287a4, built withmaturin build --releaseinto acp312 wheel and installed into the worktree venv beside torch 2.13.0+cu130.
The varena repo is UNMODIFIED — no substrate change was needed or asked for,
which is the minimalism ruling working.
Read this before the diff: the headline number is a LOSS
RTX 4070, sd1.5, 512x512, 25 steps, CFG 7.5, fp16, seed 1507, latents compared
with
torch.equal. One config, warmup, best-of-2, load 3.17 at start. Raw JSONin
benchmarks/pgw1507/.The arena is ~25% slower than the software rung at every streaming budget, and
free (1.02x) when nothing streams. Every arm bitwise identical to eager.
The cause, isolated — and the competing explanation falsified first.
Micro-benchmark on the same card:
back(4MiB)+unback(4MiB)97.8 us;back()onan already-backed range 0.2 us; DLPack export +
from_dlpack6.1 us; H2D 4 MiBpinned->arena 376.0 us vs pinned->ordinary torch buffer 368.5 us;
signature()0.1 us. So arena memory is not slower for DMA, and the whole gap is ~98 us of
page-table work per streamed leaf per forward (at 25%: ~464 page-ins/step x
98 us = 45 ms/step against a measured 60 ms/step gap). The obvious alternative —
the unback ring's CPU-side event block serialising the pipeline — was killed
before this was believed: ring depth 2/4/8/16 gives 3.56x / 3.74x / 4.07x /
3.83x. Deeper is worse.
Second cost, predicted CPU-side and confirmed to the byte: the granularity tax
is 248 MiB = 13.2%. It is charged to the planner (regions priced at their
aligned span), so
plan.resident_bytesequals bytes actually mapped andplan.fitsis exact — but the arena holds 13% less weight at the same lease.Where it wins, and what it is actually for
fadvise DONTNEED): loader 3.89 s(0.44 GB/s) vs RefillEngine->arena 3.03 s (0.57 GB/s) = 1.28x, because the
meta-init tree never allocates a torch tensor for the weights at all.
Byte-exact: 0 mismatched of 686 UNet tensors, warm and evicted.
deliverable, and the precondition for "one compiled artifact serves every
budget".
Design input for phase 3, not acted on here: a compiled cell must not page the
page table per forward at leaf granularity. Coarser cells amortise both the
98 us map and the 13.2% tax.
Identity, cycles, LoRA
Bitwise identical throughout. Three demote/promote cycles between requests:
mapped 2122 -> 66 -> 2122 MiB, signature 1->239->239, 239->477->477,
477->715->715 (advances on demote, stable across promote —
back()issignature-neutral), every resident region confirmed backed by asking the
driver, output still bitwise identical after each re-promotion. LoRA: 96
attached pairs -> 192 adapter leaves, all in the always-resident core, none ever
streamed, 36 of their base layers streaming, identical at full/25%/5%.
Three defects the card found
pipeline.deviceagain — the facade parked leaves on the host withoutstamping the handle pgw#1497's repair reads; sd1.5 died with
index is on cpu.ArenaResidency.arm()is now the production entry point and dischargesall three of
_apply_partial_stream's obligations. The repair itself neededno change: it asks the stamped handle where the model executes.
Parameter.data =across the meta boundary is refused by torch and killedthe cold-load leg. A meta parameter is replaced, not filled; every other bind
still preserves identity.
submit/waitround trips on cold adoption drained the engine queueeach time (0.83 s). One batch, both destinations: 0.37 s.
Boundaries held
compiled-offload exclusivity UNTOUCHED — it is never even consulted, because
no arena rung is on the LADDER, deliberately: a rung takes a measured price,
and the measured price is worse than
partial_stream's at every streamingbudget. It goes on the ladder when phase 3 gives it the thing it is for. No NVML,
no probe, no prefetch, no eviction heuristic.
{VRAM, RAM}pair shape is exactlywhat pgw#1497 landed — with one thing moved FORWARD, pgw#1497's own owed item: a
stated RAM half of zero now means the tail pages off disk through RefillEngine
instead of refusing.
18 CPU tests over real
nn.Moduletrees and real safetensors files, allred-armed; 58 green with pgw#1497's. ruff + mypy clean.
Named deferrals: SDXL still unmeasured (as pgw#1497 left it); serving-path rung
wiring (above); disk floors belong to va#3 Tier 1 (the evicted arms varied
3.89 s vs 15.06 s on this shared box — relative comparison only).