pgw#984 + pgw#985: the AOT recipe proves the endpoint's forward runs, and a deterministic arm decline is a typed refusal on both recipes - #514
Conversation
… and a deterministic arm decline is a typed refusal on both recipes
Both defects were MEASURED by the pgw#978 micro-mint rig inside its first
hour, by running the same machinery twice with only `MintRequest.recipe`
changed.
pgw#984 — the AOT recipe never entered the endpoint. Its measured phase table
was `load / trace_graph / seal_publish / finalize` with NO `warmup_forward`
row: `torch.export` traces the declared modules directly, so the handler was
never called. pgw#969's crash class (`ctx.slots["pipeline"]` unbound, 0.0 s
into `warmup_forward`, twice on production L40S pods) was therefore
unreachable on that recipe, and an endpoint whose forward dies on its first
request minted, sealed and published a cell fine. `mint_child.mint` now drives
the endpoint's OWN derived warm plan once, eagerly, before a byte is exported
— the same `_run_warm_job` over the same class-scoped sibling set the dynamo
recipe uses. One job, not the whole plan: the export derives its own class set.
pgw#985 — `compile_cache.begin_fleet_mint` raised
`RuntimeError: no compile targets resolved on TinyDiffusionPipeline` about a
pipeline whose `.unet` had resolved a frame earlier. Two defects in one line:
* Two computations of one fact (§1.29, th#1616). `has_compile_target` and
`apply` each scanned `cfg.targets`, and the raise reported whichever
declined under the first one's sentence — the actual decline was `apply`,
for want of CUDA. `resolve_targets` is now the one target authority and
`arming_block` the one precondition authority; the wiring fact and the
environment fact are separate sentences. Same misnaming removed from
`build` and `_compile_and_warm`.
* A bare `RuntimeError` exits 1 -> `CRASHED` -> the retryable class, for a
fact no retry can change (a second billed pod). It is now the typed
`CompileArmRefused`, which the child turns into `MintChildRefused` ->
`EXIT_REFUSED` -> terminal on attempt one, exactly as the AOT recipe has
always classified the identical condition. Warm-forward failures are
classified the same way on both recipes; resource shortfalls still
re-raise untouched so `EXIT_RESOURCE` and pgw#848's re-budget are intact.
Also: a failed mint's report carried `phase=""` in every case — `_close_phases()`
closes the open phase and the field was read after it in the same call.
RED-proved on the rig (dynamo: `crashed`/exit 1 -> `refused`/exit 2, typed;
AOT: `warmup_forward` now in the phase table) and by 8 of the 9 new rows
failing on a clean `origin/master` checkout.
…d which one arming_block owns Docstring only — the two authorities are separate functions already; this stops the target authority's own docstring from explaining the ARM fact, which is the confusion the split exists to end.
fd4f557 to
a9103f9
Compare
|
CI is green on this head; the PR gate cannot see it — GitHub's status rollup is still down. Run 31128545355 executed for real on merge head Both required contexts are present and green as check-runs on the PR head: But Head was restored to Not merging. Required-status is a real gate and the runs behind it are genuinely green — but making the PR mergeable right now would mean bypassing the gate rather than satisfying it, and the difference between "the checks passed" and "the gate observed them pass" is exactly what the gate is for. Polling for the rollup to recover. Local verification on this same head, for the record: |
1 participant
Two mint-recipe correctness defects the pgw#978 micro-mint rig found inside its first hour, by running the same machinery twice with only
MintRequest.recipechanged. Both are fixed here, RED-proved on the rig first.pgw#984 — a green AOT mint proved nothing about the cell it sealed
Measured. An AOT mint's phase table was
{'load': 4.4, 'trace_graph': 8.1, 'seal_publish': 0.9, 'finalize': 0.0}— nowarmup_forwardrow at all.torch.exporttraces the declared modules directly, somint_child._mint_aotnever called the endpoint's handler. Consequences:ctx.slots["pipeline"]unbound, 0.0 s intowarmup_forward, measured twice on production L40S pods — was unreachable on this recipe;Fix.
mint_child.mint's AOT branch now drives the endpoint's OWN derived warm plan — the same_run_warm_job, the samewarmup.planover the same class-scoped sibling set the dynamo recipe uses — once, eagerly, before a byte is exported. ONE job and not the whole plan: the export derives its own class set, so a full eager pass buys minutes and proves nothing the first forward does not. Nothing is armed for it, so it specializes no graph the export then has to trace around.execution_lane_verdict_for/_loadnow carry the endpoint INSTANCE out with the pipeline, because the handler is a method on it.Behaviour change, stated: an endpoint whose class carries no
@endpointdeclaration, or whose derived warm plan is empty, now REFUSES an AOT mint where it used to seal one. Both were already dynamo-recipe refusals.pgw#985 — a deterministic refusal was classified as a retryable crash, and named wrong
Measured. Same box, same pipeline, one recipe flag changed:
...out of
compile_cache.begin_fleet_mint— about a pipeline whose.unethas_compile_targethad resolved one frame earlier. Two defects in one line:1. Two computations of one fact (§1.29, th#1616).
has_compile_targetscannedcfg.targets;applyscanned them again in its own loop;begin_fleet_mintcalled both and reported whichever declined under the first one's sentence. What had actually declined wasapply, because the process had no CUDA — so the message ruled out the only thing it was not.compile_cache.resolve_targetsis now the ONE target authority.has_compile_target,applyandbegin_fleet_mintall read it.compile_cache.arming_blockis the ONE precondition authority — side-effect free, returns the named reasonapplylogs andbegin_fleet_mintrefuses with.applystill DECIDES (one evaluator);arming_blockonly names.buildand_compile_and_warm.2. The classification, not the message, decides the second pod. A bare
RuntimeErrorexits 1 →CRASHED→ the retryable class, for a fact no retry can change.begin_fleet_mintnow raises the typedCompileArmRefused, which the mint child turns intoMintChildRefused→EXIT_REFUSED→ terminal on attempt one — exactly what the AOT recipe has always done with the identical condition. Warm-forward failures are classified the same way on BOTH recipes now (the hub'sself_mint_abortevent already bookedphase=warmup_forwardasdeterministicwhile the worker still called itcrashed); a resource shortfall re-raises untouched, soEXIT_RESOURCEand pgw#848's re-budgeted retry are unaffected.Also: a refused or crashed mint's report carried
phase=""in every case —_close_phases()closes the open phase andMintReport.phasewas read after it in the same call.The rig proof
task rig:mint, ~20 s a cycle,--stage mint:crashed, exit 1,RuntimeError: no compile targets resolved on TinyDiffusionPipeline,phase=""refused, exit 2 (EXIT_REFUSED),not retryable,mint family='microrig' … fn='rig-generate' recipe='dynamo': TinyDiffusionPipeline owns the declared compile target(s) ['unet'] for family 'microrig', but this process cannot arm them: torch reports no CUDA device in this process,phase="load"finalize/load/seal_publish/trace_graphfinalize/load/seal_publish/trace_graph/warmup_forward(0.26 s)Tests
tests/test_mint_recipe_parity_pgw984_pgw985.py— 9 rows at the real seams, no card and no compile required:resolve_targetsis one relation: move it andhas_compile_target,applyandbegin_fleet_mintall move with it;TORCHINDUCTOR_CACHE_DIRwhere it was (gw#608's invariant, kept);EXIT_REFUSED, is not retryable, names family/function/recipe/target/why, writes no artifact, and reports the phase it died in;ctx.slots) before it exports, andwarmup_forwardlands in the phase table;MemoryErrorin the warm plan is still a resource shortfall, not a refusal.8 of the 9 fail on a clean detached checkout of
origin/master. The one that passes is the missing-target parity row — that condition was already typed on both recipes; it is the invariant, not the defect.tests/test_kernel_lane_pgw947.pydoubles updated for the 3-tuple_load.Verification
tests/3299 passed, 37 skipped, 1 xfailed (-n 4 --dist loadfile);tests_v2/21 passed.Filed, not fixed here: pgw#988 (P0)
The same rig cycle found that
origin/mastercannot adopt any AOT cell it publishes. th#1645/pgw#987 (PR #507) addedentriesto_UNBOUNDED_ENVELOPE_BLOCKS, butaot_cells._discover_innerrunsaot_serve.verifyagainst the DECLARE metadata, pre-download, and that verify requires the entries map:Reproduced on a clean detached checkout of
c2e52f5f; the rig's adopt leg is green atf029593e(before PR #507) and red atc2e52f5fand on this branch identically. Not caused by and not fixed by this PR — restoringentriesre-creates th#1645's 413, so the fix is a contract decision for that lane. Filed as pgw#988 with the measurement and three candidate fixes.Supersedes #510 — same branch, same commits. #510 stopped receiving
pull_requestevents during the 2026-08-06 Actions outage (4 close/reopen cycles and 2 pushes produced zero runs), so its status rollup stayed empty and it could not register required checks. CI is green on this branch: run 31128545355 ona9103f9c—testssuccess (12 steps, 15m48s) andfast gatessuccess (14 steps, 1m34s).