pgw#1511 + pgw#1513: verify residency before asserting it, and refuse a PROJECTED tree by name (FLEET-BLOCKING)

[PaulFidika]

pgw#1513: the eager bridge must refuse a PROJECTED tree — header too large was never about the checkpoint

FLEET-BLOCKING, and the message it replaces cost this investigation two days
pointed at poisoned volumes and truncated downloads.

THERE IS NO SHORT WRITE. Every write path was read before anything was
changed, and each is already airtight: the CAS hashes and size-checks every
object before committing it (open_writer/adopt_file/put_file), tensorfs'
materializer re-hashes each object AND the whole file and refuses a short read,
project_snapshot builds in a scratch directory and renames, and the endpoint
volume is verified in BOTH directions (fill.verify_object on read,
put_file on write-through).

WHAT IT ACTUALLY IS. A projected tree's tensor containers are ~128 B TFSSTUB1
pointer stubs; the weights live in the CAS and are read by the pgw#1380
streaming engine. The eager from_pretrained bridge is documented for "a tree
with no chunk store behind it — a bare download, a local run, a fixture", and
it reads with the stock safetensors reader, which knows nothing about stubs. It
takes the stub's first eight bytes as a header length and raises
SafetensorError: header too large.

Executed, both field shapes:

sd15-unet.safetensors:  stub 128 B on disk, names 3,400,000,000 B -> header too large
sdxl-unet.safetensors:  stub 129 B on disk, names 68,000,000,000 B -> header too large

A stub is a FIXED SIZE regardless of the model behind it. That is why a 3.4 GB
and a 68 GB checkpoint failed byte-identically across two endpoints and two
volumes — a coincidence no real truncation produces, and the fact that should
have named this on day one.

WHY IT STARTED AT pgw#1490. Before it, tree_for required
ModelBinding.manifest_digest, which has never had a sender, so it ALWAYS
raised — no v2 dispatch ever reached the loader. Making the dispatch resolvable
made the eager bridge reachable for the first time, so the one reader that
never got a stub-aware path met the tree format it was always going to receive.

WHAT THIS CHANGES

1. THE BRIDGE REFUSES, BY NAME. ProjectedTreeNotStreamable names the member,
   the bytes on disk, the bytes the stub stands for, and — the part that stops
   the next two-day hunt — WHICH of resolve_projection's three silent Nones
   fired, with the repair for each: wrong parent directory, missing
   refs/objects, or a missing snapshot:<key> pin. Reaching the bridge with a
   projected tree means the streaming engine declined it, and until now that
   fact was unobservable.
2. announce_resident REFUSES A TREE NO ENGINE CAN BIND. There is one
   byte-perfect state that passes full digest verification and still cannot be
   served: the tree is present and its manifest pin is absent. Answering
   already_resident there strands the pod in exactly the failure above.
   Answering False sends it through ensure_local, which re-pins and — because
   _tree_matches passes — returns the SAME tree. A missing pin is REPAIRED,
   NOT RE-DOWNLOADED; no bytes move.

WHAT THIS DELIBERATELY DOES NOT DO. No hydration. Filling tensor bytes into a
real file so from_pretrained can mmap it is ruled out by DESIGN-RULINGS
("serving path never materializes tensor bytes") and would delete the very
signal that reveals the bug, leaving a silent perf/VRAM regression instead of a
loud stop. The detector branches on read_stub() BEFORE any parse, never on a
SafetensorError — pgw#1308's two wrong callers both inferred a fact about the
WORLD from a fact about a READER, and "on SafetensorError, hydrate and retry"
is that mistake wearing a repair. Single-object pointer routes are dead by
arithmetic (MAX_OBJECT_SIZE = 64 MiB; real containers are 4, 18, 51, 63
objects), so there is ONE mechanism, not two.

HONEST LIMIT, stated because a re-pin depends on it: I have NOT reproduced the
field condition on the real dispatch path. Probed after a real boot
materialization, the path the dispatch hands the loader is
<cas>/snapshots/<key> with pin present, resolve_projection OK, store_for
OK, engine WOULD bind — the healthy control. So the pin-missing state fixed
here is a real hole reachable when a tree outlives its pin, but it is not
proven to be the field cause. What this change guarantees is that the next
occurrence reports which condition failed in one line instead of lying about
the checkpoint.

Red/green, $0, no GPU, no pod:

both field shapes refused by name, loader never called | pass | RED: SafetensorError,
| | header too large (x2)
the fixed-size-stub signature across a 20x model | pass | —
a MATERIALIZED tree still takes the eager bridge | pass | — (the local-dev control)
a tree with a MISSING PIN is not answered as resident | pass | —

The red arm restores the landed stub-blind bridge and flips a CONDITION, never
cuts lines. Neighbours green: test_boot_materialize, test_weight_position,
test_model_residency, test_store_corruption_pgw1283 — 35 passed.
ruff check src/gen_worker clean; whole-tree mypy clean (614 files, run with
MYPYPATH so it does not resolve against the canonical checkout's stale src).

Credit: the se#790/anima lane's local repro proved all three
resolve_projection conditions produce this symptom independently while the
healthy control binds, which is what turned "make a reader stub-aware" into
"find out why the engine did not bind" — and their pgw#1308 scar tissue is why
this refuses instead of repairing.

---

This PR carries two fixes. First commit is pgw#1511 (a tree that EXISTS is not a tree that is RESIDENT — verify before asserting, quarantine when it fails); second is pgw#1513 above, which is the one that gates the rentals.

Needs pgw#1073 (pgw#1512) for a green base — pgw master fast gates is red for every lane on an unrelated ruff F541 and on nine unwired gguf_torch callables from pgw#1498.

🤖 Generated with Claude Code

[PaulFidika]

Merging with a TARGETED --admin. The evidence is CI's own output, not my local run.

fast gates on this branch reports exactly one failing step:

$ gh api .../actions/jobs/96277672742 --jq '[.steps[]|select(.conclusion=="failure")]|length'
1
$ ... --jq '.steps[]|select(.conclusion=="failure")|.name'
pgw#931 guard - config reads outside the pipeline

That guard is red on master itself, from pgw#1491's ce4e8eba. Verified on a clean base checkout with none of this branch's content: 5 problems, all in cli/compile.py and cli/credentials.py. This PR touches models/projection.py, models/store.py, serving/context.py and two test files — no cli/ file at all, so it cannot produce a config-read violation.

Every other guard passes on this branch, run locally after the final rebase: unreached-surface, incident-test-names, mypy ratchet, HTTP timeouts, settings writers, materialization hatch, ruff. Whole-tree mypy clean (616 files). The three tests failures were separately attributed to master by executing the extra one (test_release_derive.py::test_derive_runs_in_a_release_env_with_no_top_level_torchcg_or_tensorfs) on a clean master checkout: it fails there too.

This is the legitimate half of the pgw#1521 asymmetry — "red because of somebody else's problem on master" — established by execution against the base rather than asserted, which is the standard that issue now records. It does not fix master: pgw#1491's lane still owns the config-reads classification, and every other lane hits it until they land it.

This PR has burned five CI rounds, every one of them on another lane's master-red.