pgw#606/th#938: Slot fns never boot-setup from the image-baked code default#353
Merged
Merged
Conversation
…efault — hub-stamped release binding is the only hub-connected setup source (Hot DesiredInstance / dispatch _effective_spec)
PaulFidika
added a commit
that referenced
this pull request
Jul 21, 2026
…idency reconcile, slot binding precedence, CAS multiwriter, discovery lock, cancel/ backpressure, boot smoke) Harness extracted per design: hub_double.py generalizes FakeScheduler from test_worker_grpc_e2e.py (the only double, at the true hub gRPC boundary); blob_host.py is a real blake3-verified HTTP blob server; toy_endpoints.py consolidates toy fixtures (echo/stream/GPU-slot-yield/Hub-bound/Slot-bound/ billable); subprocess_runner.py generalizes the gw#591 boot-smoke pattern. P3 absorbs PR #353 (pgw#606/th#938) and PR #307 (pgw#583) as real hub-double dispatches (no mocking of gen_worker internals, unlike the original unit tests) - revert-verified red against lifecycle.py's pre-fix state, restored clean. P1/P2/P4/P5/P6/P7 absorb assertions from test_worker_grpc_e2e.py, test_shared_cas_root_multiwriter.py, and test_boot_smoke_gw591.py. Found + fixed in the harness itself: WorkerHarness must set the TENSORHUB_CACHE_DIR env var (gen_worker.models.cache_paths reads the process-wide get_settings() singleton, not the per-worker Settings instance) - passing tensorhub_cache_dir= to load_settings() alone silently no-ops, and every hub-double test defaults to a fresh per-call temp dir so downloaded bytes never leak between tests. 40 passed, 1 skipped (pgw#605 heartbeat: proto fields not generated yet) in ~33s local. Existing test files not touched or deleted (Phase 3).
PaulFidika
added a commit
that referenced
this pull request
Jul 21, 2026
…egacy deletion (166→40 files) (#359) * th#960/pgw#609 Phase 2: tests/harness/ + P1-P7 (stream lifecycle, residency reconcile, slot binding precedence, CAS multiwriter, discovery lock, cancel/ backpressure, boot smoke) Harness extracted per design: hub_double.py generalizes FakeScheduler from test_worker_grpc_e2e.py (the only double, at the true hub gRPC boundary); blob_host.py is a real blake3-verified HTTP blob server; toy_endpoints.py consolidates toy fixtures (echo/stream/GPU-slot-yield/Hub-bound/Slot-bound/ billable); subprocess_runner.py generalizes the gw#591 boot-smoke pattern. P3 absorbs PR #353 (pgw#606/th#938) and PR #307 (pgw#583) as real hub-double dispatches (no mocking of gen_worker internals, unlike the original unit tests) - revert-verified red against lifecycle.py's pre-fix state, restored clean. P1/P2/P4/P5/P6/P7 absorb assertions from test_worker_grpc_e2e.py, test_shared_cas_root_multiwriter.py, and test_boot_smoke_gw591.py. Found + fixed in the harness itself: WorkerHarness must set the TENSORHUB_CACHE_DIR env var (gen_worker.models.cache_paths reads the process-wide get_settings() singleton, not the per-worker Settings instance) - passing tensorhub_cache_dir= to load_settings() alone silently no-ops, and every hub-double test defaults to a fresh per-call temp dir so downloaded bytes never leak between tests. 40 passed, 1 skipped (pgw#605 heartbeat: proto fields not generated yet) in ~33s local. Existing test files not touched or deleted (Phase 3). * th#960/pgw#609 Phase 2: P8 (convert publish contract) + P9 (result upload metrics), plus ruff cleanup + a hub_double(file_base_url=) knob P9 needs P8 (tests/convert/test_p8_convert_publish_contract.py, hermetic fake_hub): dtype-mismatch cast honesty (pgw#589/th#901, condensed from test_publish_as_is_dtype.py), classifier corpus table (pgw#593), and an xfail(strict=True) test-first pin for pgw#566 (open bug: normalize_adapter _state_dict only passes unet_config to converters with a NAMED parameter - diffusers' real SDXL lora_state_dict takes **kwargs only, so the SGM block remap silently never runs). pgw#569's W8A8 verifier ULP gate is a documented skip - no factored-out comparator to test hermetically without a real w8a8 artifact round trip. P9 (tests/test_p9_result_upload_metrics.py): inline vs blob_ref by size alone over a real hub-double + real local media-upload dedup sink (no S3 part scripting, matching test_media_upload_owner.py's pattern); JobMetrics typed usage survives both wire forms. Needed hub_double(file_base_url=) so the sink URL rides the FIRST HelloAck instead of a racy post-connect mutate. ruff clean (unused imports, one E731 lambda->named helper). mypy: src/gen_worker clean (untouched); new tests/harness + test files carry the same untyped- fixture-param convention as the rest of tests/ (mypy only gates src/gen_worker per CI, tests/ was never in scope). Full new suite (P1-P9 + P8's own file): 50 passed, 1 skipped (pgw#605, proto fields not generated), 1 xfailed (pgw#566, fix open) in ~31s local. * th#960/pgw#609 Phase 3: delete 4 test files fully superseded by P3/P4/P8 test_slot_boot_precedence_th938.py, test_model_slot_identity_gate.py -> superseded by test_p3_slot_binding_precedence.py (same assertions, over a real hub-double boundary instead of monkeypatched Executor.store internals). test_shared_cas_root_multiwriter.py -> superseded by test_p4_cas_multiwriter_integrity.py (same 3 tests reproduced verbatim + 1 new row). tests/convert/test_publish_as_is_dtype.py -> superseded by tests/convert/test_p8_convert_publish_contract.py (same 3 core tests against the same fake_hub). Deletions only, no other changes. Full deletion manifest (all 166 files audited, reasoning for what's NOT deleted) is in the th#960/pgw#609 tracker checkpoint - the bulk of the pre-existing suite tests production surface (CLI, LoRA runtime, quantization lanes, VRAM/RAM admission, media/upload, Slot/discovery SDK, tests/convert producer suite) that no P-test covers, so it stays pending explicit direction on the larger absorb-or-accept-loss effort that would be needed to delete it safely. * pgw#609/th#960 Phase 4: rewire CI to one suite, one lane Manual dispatch + PR-into-master only (drafts/doc-only paths run nothing); drop the on-push trigger. Same gates (mypy gating, ruff non-gating per debt note, HTTP-timeout guard, llama-server, uv build); pytest targets the whole tests/ dir so P-suite additions land automatically. * th#960/pgw#609 Phase 2b: absorb ~26 incident-pinned rows into P1/P2/P3/P5/ P6/P8/P9 + 2 new consolidated files, ahead of the bounded aggressive deletion sweep Bucket-level triage per the coordinator's ruling (Paul's directive: minimal greenfield set, ~92% removal). Absorbed only distinct bug-classes carrying a real pgw#/gw# incident number or a revert-catches-it story; untagged "tested thoroughly once" coverage goes without absorption (git history is the archive). P1 (+6): auth-rejection exit/retry-within-window, permanent-precondition fast-exit, hello-ack-deadline reconnect, not-leader redirect, hub-restart reconnect - absorbed from test_worker_grpc_e2e.py (#372) ahead of its deletion. P2 (+3): setup-failure FnUnavailable+recovery, host-RAM-failure wire ordering (th#807) from test_worker_grpc_e2e.py; corrupt-load quarantine+ retry-once (gw#408, J17 flood) from test_snapshot_corruption.py. P3 (+1): bare-ref prefetch binding resolution (#377) from test_executor_prefetch_binding.py. P5 (+1): heavy-dep stub contract (pgw#506) from test_discovery_heavy_deps.py. P6 (+1): SendQueue capacity-generation fencing from test_worker_grpc_e2e.py. P8 (+5): diffusers root-allinone skip (e2e J7 ENOSPC), standalone-component canonical-weight selection (gw#426), transformers sharded-index onnx/pickle exclusion, gguf explicit-quant refusal, non-safetensors-format refusal family - from test_classifier.py ahead of its deletion. P9 (+1): token-bound-owner routing (J19 run34) from test_media_upload_owner.py. New consolidated files (coordinator-authorized, one per lane with no P-test home): tests/test_quantization_lanes.py (W8A8/W4A4/FP8 contract + gw#409 promote-device-integrity, from test_w8a8.py/test_w4a4.py/ test_fp8_and_emergency_loading.py/test_promote_device_integrity.py) and tests/convert/test_convert_producer_contract.py (clone concurrency gw#442, disk preflight gw#462, download-skip bank th#592, publish resilience gw#462, from the tests/convert/ producer suite). Also reverted a premature local-only Phase 4 CI commit (c39efb6, from a prior session before an interruption) - never pushed, out of THIS issue's authorized scope ("do not touch CI yet"); preserved in history via revert, not discarded. Full new-suite gates: mypy clean, ruff clean, each new/changed file green in isolation. Deletion commit (the ~125 files these rows supersede) follows as a separate, deletions-only commit per the coordinator's mechanics. * th#960/pgw#609 Phase 3b: aggressive deletion sweep — 125 files (32,618 LoC) superseded or absorbed per the coordinator's ruling Every file here is either fully superseded by test_quantization_lanes.py / test_convert_producer_contract.py, absorbed as rows into P1/P2/P3/P5/P6/P8/ P9 (chaos b8274a1), or carries no incident pin (git history is the archive, per the coordinator's bound). Full manifest, bucket-by-bucket reasoning, and the dangling-import sweep are in the th#960/pgw#609 tracker Checkpoint 6, pushed before this commit. Deletions only, no other changes. Foreign-WIP clusters untouched: compile_cache/executor_adopt/fleet_cells (this issue's own claim-checkpoint note) and test_disk_telemetry_pgw610.py (landed on chaos by another lane during this session). Final state: 14 test files (3,293 LoC) + tests/harness/ (6 files) + conftest.py x2 + fake_hub.py, target was ~12-15 files per the coordinator.
PaulFidika
added a commit
that referenced
this pull request
Jul 21, 2026
…4 + wan family) (#363) * gw#587/gw#605: pin — a fresh Executor (no setup yet) advertises pre-load candidate cell lookups Unit-level the candidates work (records are eagerly created at construction); the LIVE gap (4 consecutive boots never received the stored cell — every boot re-minted) is therefore in the candidate->StateDelta->release-config-> attach chain, not in cell_lookups() itself. Forensics + eliminated theories in tracker gw#605. * compile-cache: platform org _system -> root (lockstep with tensorhub th#940 chaos 0e4f6b1e) system_repo now derives root/family-<f>; ref parsing, self-mint publisher owner, docs, proto contract, and test fixtures move in lockstep. Only the org segment changes — repo/flavor grammar is byte-identical. * gw#608: disable the AOT autograd cache in the compile path — cell portability requires the FX cache as the lookup surface; pack asserts its FX entries before publish ROOT CAUSE (cracked dry, three same-key captures from three pods): the AOTAutogradCache key hashes fx_kwargs[get_decomp_fn] via the function REPR — which embeds the process memory address (ASLR), so AOT keys can NEVER match across processes. inductor/fxgraph entries were bit-identical across all pods (8/8); inductor/aotautograd keys had zero overlap. On the consumer, the AOT-layer miss recompiles without consulting the portable on-disk FX entries: cache_hits=0, cache_misses=8 on two independent hosts against a byte-good cell (fail-closed held — never eager). Fix: _disable_aot_autograd_cache() (env TORCHINDUCTOR_AUTOGRAD_CACHE=0 + torch._functorch.config.enable_autograd_cache=False) at capture_env/ seed_env AND apply() — symmetric for producer capture and consumer seeding. Lookups collapse to FxGraphCache (proven portable); cells shrink (~50% was AOT payload); fresh processes pay a cheap AOT re-analysis, not an inductor recompile. Hardening (coordinator ruling): a minting boot proves its EXECUTION — finish_fleet_mint now also proves its ARTIFACT: refuse to publish a capture with zero fxgraph entries, and (when the proof supplies its compiled-graph count via finalize_self_mint(expected_graphs=pipe_misses)) a partial capture. Revert-turns-red tests for both. (5 pre-existing chaos test failures reproduced on untouched HEAD.) * gw#608: adopt-test fake warmup writes the fxgraph store the artifact assertion requires (one entry per miss) — the 0.40.4 CI failure was this un-updated fake, not the capture path * th#960/pgw#609 Phase 2: tests/harness/ + P1-P7 (stream lifecycle, residency reconcile, slot binding precedence, CAS multiwriter, discovery lock, cancel/ backpressure, boot smoke) Harness extracted per design: hub_double.py generalizes FakeScheduler from test_worker_grpc_e2e.py (the only double, at the true hub gRPC boundary); blob_host.py is a real blake3-verified HTTP blob server; toy_endpoints.py consolidates toy fixtures (echo/stream/GPU-slot-yield/Hub-bound/Slot-bound/ billable); subprocess_runner.py generalizes the gw#591 boot-smoke pattern. P3 absorbs PR #353 (pgw#606/th#938) and PR #307 (pgw#583) as real hub-double dispatches (no mocking of gen_worker internals, unlike the original unit tests) - revert-verified red against lifecycle.py's pre-fix state, restored clean. P1/P2/P4/P5/P6/P7 absorb assertions from test_worker_grpc_e2e.py, test_shared_cas_root_multiwriter.py, and test_boot_smoke_gw591.py. Found + fixed in the harness itself: WorkerHarness must set the TENSORHUB_CACHE_DIR env var (gen_worker.models.cache_paths reads the process-wide get_settings() singleton, not the per-worker Settings instance) - passing tensorhub_cache_dir= to load_settings() alone silently no-ops, and every hub-double test defaults to a fresh per-call temp dir so downloaded bytes never leak between tests. 40 passed, 1 skipped (pgw#605 heartbeat: proto fields not generated yet) in ~33s local. Existing test files not touched or deleted (Phase 3). * th#960/pgw#609 Phase 2: P8 (convert publish contract) + P9 (result upload metrics), plus ruff cleanup + a hub_double(file_base_url=) knob P9 needs P8 (tests/convert/test_p8_convert_publish_contract.py, hermetic fake_hub): dtype-mismatch cast honesty (pgw#589/th#901, condensed from test_publish_as_is_dtype.py), classifier corpus table (pgw#593), and an xfail(strict=True) test-first pin for pgw#566 (open bug: normalize_adapter _state_dict only passes unet_config to converters with a NAMED parameter - diffusers' real SDXL lora_state_dict takes **kwargs only, so the SGM block remap silently never runs). pgw#569's W8A8 verifier ULP gate is a documented skip - no factored-out comparator to test hermetically without a real w8a8 artifact round trip. P9 (tests/test_p9_result_upload_metrics.py): inline vs blob_ref by size alone over a real hub-double + real local media-upload dedup sink (no S3 part scripting, matching test_media_upload_owner.py's pattern); JobMetrics typed usage survives both wire forms. Needed hub_double(file_base_url=) so the sink URL rides the FIRST HelloAck instead of a racy post-connect mutate. ruff clean (unused imports, one E731 lambda->named helper). mypy: src/gen_worker clean (untouched); new tests/harness + test files carry the same untyped- fixture-param convention as the rest of tests/ (mypy only gates src/gen_worker per CI, tests/ was never in scope). Full new suite (P1-P9 + P8's own file): 50 passed, 1 skipped (pgw#605, proto fields not generated), 1 xfailed (pgw#566, fix open) in ~31s local. * th#960/pgw#609 Phase 3: delete 4 test files fully superseded by P3/P4/P8 test_slot_boot_precedence_th938.py, test_model_slot_identity_gate.py -> superseded by test_p3_slot_binding_precedence.py (same assertions, over a real hub-double boundary instead of monkeypatched Executor.store internals). test_shared_cas_root_multiwriter.py -> superseded by test_p4_cas_multiwriter_integrity.py (same 3 tests reproduced verbatim + 1 new row). tests/convert/test_publish_as_is_dtype.py -> superseded by tests/convert/test_p8_convert_publish_contract.py (same 3 core tests against the same fake_hub). Deletions only, no other changes. Full deletion manifest (all 166 files audited, reasoning for what's NOT deleted) is in the th#960/pgw#609 tracker checkpoint - the bulk of the pre-existing suite tests production surface (CLI, LoRA runtime, quantization lanes, VRAM/RAM admission, media/upload, Slot/discovery SDK, tests/convert producer suite) that no P-test covers, so it stays pending explicit direction on the larger absorb-or-accept-loss effort that would be needed to delete it safely. * pgw#610: proto — measured per-tier DiskUsageReport on StateDelta (lockstep with th#962) * pgw#610: measured per-tier disk telemetry in every StateDelta (th#962 lockstep) - statvfs on the real mounts (CAS root = container tier; ismount-guarded fill source volume when attached; NFS tier reserved) — never a declared size - safely-reclaimable = ref-index bytes of DISK-tier refs inactive AND not in the desired set (the disk-GC LRU's eligible set); reuses ref-index bytes, no tree rescans - capacity_generation bumps only on a measured (64 MiB-quantized) shape change; rides Hello.state and every StateDelta; 30s periodic refresher is edge-suppressed * pgw#609/th#960 Phase 4: rewire CI to one suite, one lane Manual dispatch + PR-into-master only (drafts/doc-only paths run nothing); drop the on-push trigger. Same gates (mypy gating, ruff non-gating per debt note, HTTP-timeout guard, llama-server, uv build); pytest targets the whole tests/ dir so P-suite additions land automatically. * Revert "pgw#609/th#960 Phase 4: rewire CI to one suite, one lane" This reverts commit c39efb69b82d967393bd3f3243ce328d954380f7. * th#960/pgw#609 Phase 2b: absorb ~26 incident-pinned rows into P1/P2/P3/P5/ P6/P8/P9 + 2 new consolidated files, ahead of the bounded aggressive deletion sweep Bucket-level triage per the coordinator's ruling (Paul's directive: minimal greenfield set, ~92% removal). Absorbed only distinct bug-classes carrying a real pgw#/gw# incident number or a revert-catches-it story; untagged "tested thoroughly once" coverage goes without absorption (git history is the archive). P1 (+6): auth-rejection exit/retry-within-window, permanent-precondition fast-exit, hello-ack-deadline reconnect, not-leader redirect, hub-restart reconnect - absorbed from test_worker_grpc_e2e.py (#372) ahead of its deletion. P2 (+3): setup-failure FnUnavailable+recovery, host-RAM-failure wire ordering (th#807) from test_worker_grpc_e2e.py; corrupt-load quarantine+ retry-once (gw#408, J17 flood) from test_snapshot_corruption.py. P3 (+1): bare-ref prefetch binding resolution (#377) from test_executor_prefetch_binding.py. P5 (+1): heavy-dep stub contract (pgw#506) from test_discovery_heavy_deps.py. P6 (+1): SendQueue capacity-generation fencing from test_worker_grpc_e2e.py. P8 (+5): diffusers root-allinone skip (e2e J7 ENOSPC), standalone-component canonical-weight selection (gw#426), transformers sharded-index onnx/pickle exclusion, gguf explicit-quant refusal, non-safetensors-format refusal family - from test_classifier.py ahead of its deletion. P9 (+1): token-bound-owner routing (J19 run34) from test_media_upload_owner.py. New consolidated files (coordinator-authorized, one per lane with no P-test home): tests/test_quantization_lanes.py (W8A8/W4A4/FP8 contract + gw#409 promote-device-integrity, from test_w8a8.py/test_w4a4.py/ test_fp8_and_emergency_loading.py/test_promote_device_integrity.py) and tests/convert/test_convert_producer_contract.py (clone concurrency gw#442, disk preflight gw#462, download-skip bank th#592, publish resilience gw#462, from the tests/convert/ producer suite). Also reverted a premature local-only Phase 4 CI commit (c39efb6, from a prior session before an interruption) - never pushed, out of THIS issue's authorized scope ("do not touch CI yet"); preserved in history via revert, not discarded. Full new-suite gates: mypy clean, ruff clean, each new/changed file green in isolation. Deletion commit (the ~125 files these rows supersede) follows as a separate, deletions-only commit per the coordinator's mechanics. * th#960/pgw#609 Phase 3b: aggressive deletion sweep — 125 files (32,618 LoC) superseded or absorbed per the coordinator's ruling Every file here is either fully superseded by test_quantization_lanes.py / test_convert_producer_contract.py, absorbed as rows into P1/P2/P3/P5/P6/P8/ P9 (chaos b8274a1), or carries no incident pin (git history is the archive, per the coordinator's bound). Full manifest, bucket-by-bucket reasoning, and the dangling-import sweep are in the th#960/pgw#609 tracker Checkpoint 6, pushed before this commit. Deletions only, no other changes. Foreign-WIP clusters untouched: compile_cache/executor_adopt/fleet_cells (this issue's own claim-checkpoint note) and test_disk_telemetry_pgw610.py (landed on chaos by another lane during this session). Final state: 14 test files (3,293 LoC) + tests/harness/ (6 files) + conftest.py x2 + fake_hub.py, target was ~12-15 files per the coordinator. * pgw#610: lint — typed enum cast + drop unused import * pgw#610: CONTRACT.md — document StateDelta.disk_usage / DiskUsageReport * pgw#610: guard drain's disk-report cancel with getattr (stubbed Lifecycles skip __init__, same convention as _cancel_residency_reconcile) * gw#608 residual: AOT-cache disable must be process-global — torch 2.13 config user overrides are thread-local ContextVars Live disproof (0.40.5, B200, 2026-07-21): mint capture still packed 8 ASLR-keyed aotautograd entries and the store-served sibling still failed (warmups=1, cache_hits=0, cache_misses=8) — the 0.40.4 disable ran on the arming thread; the warmup compile thread saw the default True. The env var is read once at torch import (env_name_force), so post-import it is inert. Fix: entrypoint sets TORCHINDUCTOR_AUTOGRAD_CACHE=0 before any torch import (binds all threads + compile-worker subprocesses); _disable_aot_autograd_cache additionally mutates the installed config entry's env_value_force (process- global) for already-imported-torch embedders. Cross-thread revert-turns-red test added (fails on the 0.40.5 shape, passes with the fix). Dry evidence: fxgraph keys 8/8 identical across ALL five banked packs (0.40.1 x3 / 0.40.3 / 0.40.5); aotautograd level-2 keys 0/8 everywhere. samples/selfmint-proof/gw0405-run/. * th#964: family lane policy twin — bare conv-UNet (sd1/sd2/sdxl) bindings AUTO-pick scale-free #fp8 locally gen_worker.models.ladder gains family_root (modelfamily.Root twin) and the conv-UNet w8a8 exclusion table; a maybe_rebind_family_fp8 fold (hub-parity AUTO pick over the resolve's sibling_flavors, sm_89+ only, bf16 stays the sub-floor default; pins untouched) runs in the local CLI resolve ahead of the GGUF fold over one shared resolve. WorkerResolvedRepo carries the resolve's model_family (slot-declared family as fallback); select_gguf no longer lets an AUTO-ineligible w8a8 row suppress the GGUF pick. Shared vectors: none in this repo; no format change. * gw#613/th#965: universal app-level heartbeat — layer 2 of the 3-layer liveness contract (0.41.0) Hello declares heartbeat_interval_ms=30000; the pgw#610 disk-report task is promoted to the beat: it force-re-sends the full StateDelta every 30s from the asyncio event loop (the control loop that owes progress, never a detached thread), starts at startup() entry so boot hangs are covered, and keeps beating through drain until the stream closes. Edge-triggered deltas still flow as before; the beat is the same message re-sent unchanged. Contract §3 rewritten to the 3-layer model (lockstep tensorhub c03a531d). Tests: hello cadence, force-send bypasses edge suppression, beats flow while a startup coroutine is parked (the gw#612 shape), drain keeps beating until close. * gw#613 retune: 10s beat x 6 misses; disk stats measured every 30s riding every beat HEARTBEAT_INTERVAL_MS 30000 -> 10000 (hub reaps at 6 misses, ~60s). The pgw#610 statvfs/ref-index scan keeps its 30s measurement cadence behind a TTL cache so the 10s beat never turns it into a hot loop; the report still rides every StateDelta. Contract §3 event-loop discipline clause added — already true in 0.41.0: setup/warmup/compile/residency/GC all run off-loop (executor._to_thread_complete / asyncio.to_thread). * gw#612: gate self-mint publish on full capture coverage; finalize phase; run-26 root cause corrected (0.42.0) Run 26's filed post-seal_publish hang is disproven: the worker completed setup and advertised readiness; the wedge was the hub's singular compile fence treating the 2-lane record's same-identity self-attested targets as ambiguous (tensorhub lockstep fix). Real worker defect fixed: a shared mint capture missing an unexercised mandatory sibling's graphs was still published as the family cell, bricking every adopting boot (gw#611 qwen hits=1/misses=1). finalize_self_mint packs only; the executor publishes iff every capture sharer proved in, else withholds typed+loud. New 'finalize' activity phase covers the post-proof tail to readiness. * gw#613: soften run-26 narrative — app-silent, not proven-hung (gw#612 root cause: healthy-idle, hub fence starvation) * gw#613: marco_polo_wedge — event-loop-blocking liveness probe for th#965 layer-2 drills * gw#611: credit AOT-layer hits in the adopt warmup proof; calls= in the fail-closed detail; cross-process portability repro Measured mechanism for th#954's release-bricking 0/0: bundled AOT cache serves a fresh-process adopt with fxgraph counters fully silent, so a HEALTHY serving cell read cache_hits=0/cache_misses=0 and fail-closed. inductor_counters now surfaces aot_cache_hit/aot_cache_miss and the guard wrapper counts AOT hits as serving evidence (AOT stays pinned off per gw#608; a config regression now degrades to a proven boot, not a brick). Fail-closed detail carries calls= to split orphaned-wrapper (calls=0) from counter-blind serving (calls>0). New tests: real mint->pack->fresh- subprocess adopt asserts >=1 FX hit; bundled-AOT 0/0 mechanism pin; guard credits AOT hits (red on revert). Hub lockstep: tensorhub chaos 9e7dca8e quarantines cells failing their own adopt proof. * th#767: gen_worker.families.wan — WanDefaults (animegen lane) Second family after SdxlDefaults: steps/guidance/guidance_2/max_guidance for the Wan 2.2 A14B MoE envelope. Registered under tensorhub's canonical architecture root "wan22" (not a Compile(family=) shape-grid string) so repo-metadata PUT validates against the matching tensorhub schema (companion wan22.schema.json on tensorhub chaos). Lets the wan-2.2 endpoint's T2V function migrate to the Slot model and give AnimeGen-T2V its own distilled recipe (8 steps, guidance=1.0, guidance_2=1.0) via repo metadata instead of branching on checkpoint identity. * gw#614: synthesized media-modality warmup coverage (union family cells publish) + on_hello_ack model-set-diff cancel (A) _run_synthesized_warmup coverage pass: compile-target objects still unexercised after the planned jobs get media VARIANTS of the same base payloads (warmup.media_variants: one optional image/audio field filled with a generated asset, nothing else drifts) — input-routed sibling lanes (qwen edit) mint into the shared capture, so the gw#612 gate publishes the union cell instead of withholding forever; adopts prove the sibling lane instead of arming unproven. (B) th#961 defense in depth: a HelloAck whose semantic model set (resolutions+disk_refs+snapshots+hot) matches the running reconcile's target no longer cancels it (in-flight self_mint_compile survives benign plan rewrites); changed sets cancel as before. * gw#615: fix event-loop-blocking disk telemetry — the 0.40.7 LTX post-seal_publish hang pgw#610/th#962's disk_usage_report() ran statvfs()/stat() synchronously inline in Lifecycle._state_delta(), which many hot paths call directly with no event-loop offload — including right after a self-mint's seal_publish. The provider-attached VOLUME fill-source is a network-backed mount that can stall for minutes under load (exactly what a self-mint's weight download + cell pack produce). A stalled statvfs() there freezes the whole event loop: every StateDelta, RunJob dispatch, and even the th#965 heartbeat (same loop) stops until the syscall returns. ModelStore.disk_usage_report() now only reads a cache; the actual measurement (_measure_disk_usage_report) runs exclusively via refresh_disk_usage_report(), off-loop through asyncio.to_thread. Lifecycle kicks that refresh as a fire-and-forget background task (never awaited inline), gated to the existing ~30s TTL, so a stalled mount delays only its own telemetry freshness — never any send, dispatch, or heartbeat. Root-caused via git diff v0.40.5..v0.40.7 (executor.py/lifecycle.py) plus gw#612's hub-log forensics for the qwen case: gw#612 itself was hub-side (dispatch-fence starve, corrected root cause, no worker hang) and its worker-side companion fix addresses a different defect (poisoned family-cell publish on the NEXT boot). This is a distinct, genuinely-new gen-worker regression introduced by a70e7c9 (wave-2 telemetry), reproduced locally with no GPU via a stalled-statvfs monkeypatch. Tests: tests/test_disk_telemetry_pgw610.py (new test_disk_usage_report_never_measures_only_reads_the_cache, test_state_delta_never_blocks_event_loop_on_a_stalled_mount; existing tests updated to the cache/refresh split) + tests/test_heartbeat_gw613.py (new test_beats_flow_while_disk_measurement_is_stalled; existing TTL test updated to the fire-and-forget refresh). All revert-turns-red verified. Full suite 458 passed/5 skipped; ruff+mypy clean. * release: gen-worker 0.42.0 (gw#615 serving-safety fix + gw#611/612/614 + wan family)
PaulFidika
added a commit
that referenced
this pull request
Jul 22, 2026
…621 progress registry, gw#619 hw-unsuitable report, ie#522 wan shift, gw#585/608 lanes (#367) * gw#587/gw#605: pin — a fresh Executor (no setup yet) advertises pre-load candidate cell lookups Unit-level the candidates work (records are eagerly created at construction); the LIVE gap (4 consecutive boots never received the stored cell — every boot re-minted) is therefore in the candidate->StateDelta->release-config-> attach chain, not in cell_lookups() itself. Forensics + eliminated theories in tracker gw#605. * compile-cache: platform org _system -> root (lockstep with tensorhub th#940 chaos 0e4f6b1e) system_repo now derives root/family-<f>; ref parsing, self-mint publisher owner, docs, proto contract, and test fixtures move in lockstep. Only the org segment changes — repo/flavor grammar is byte-identical. * gw#608: disable the AOT autograd cache in the compile path — cell portability requires the FX cache as the lookup surface; pack asserts its FX entries before publish ROOT CAUSE (cracked dry, three same-key captures from three pods): the AOTAutogradCache key hashes fx_kwargs[get_decomp_fn] via the function REPR — which embeds the process memory address (ASLR), so AOT keys can NEVER match across processes. inductor/fxgraph entries were bit-identical across all pods (8/8); inductor/aotautograd keys had zero overlap. On the consumer, the AOT-layer miss recompiles without consulting the portable on-disk FX entries: cache_hits=0, cache_misses=8 on two independent hosts against a byte-good cell (fail-closed held — never eager). Fix: _disable_aot_autograd_cache() (env TORCHINDUCTOR_AUTOGRAD_CACHE=0 + torch._functorch.config.enable_autograd_cache=False) at capture_env/ seed_env AND apply() — symmetric for producer capture and consumer seeding. Lookups collapse to FxGraphCache (proven portable); cells shrink (~50% was AOT payload); fresh processes pay a cheap AOT re-analysis, not an inductor recompile. Hardening (coordinator ruling): a minting boot proves its EXECUTION — finish_fleet_mint now also proves its ARTIFACT: refuse to publish a capture with zero fxgraph entries, and (when the proof supplies its compiled-graph count via finalize_self_mint(expected_graphs=pipe_misses)) a partial capture. Revert-turns-red tests for both. (5 pre-existing chaos test failures reproduced on untouched HEAD.) * gw#608: adopt-test fake warmup writes the fxgraph store the artifact assertion requires (one entry per miss) — the 0.40.4 CI failure was this un-updated fake, not the capture path * th#960/pgw#609 Phase 2: tests/harness/ + P1-P7 (stream lifecycle, residency reconcile, slot binding precedence, CAS multiwriter, discovery lock, cancel/ backpressure, boot smoke) Harness extracted per design: hub_double.py generalizes FakeScheduler from test_worker_grpc_e2e.py (the only double, at the true hub gRPC boundary); blob_host.py is a real blake3-verified HTTP blob server; toy_endpoints.py consolidates toy fixtures (echo/stream/GPU-slot-yield/Hub-bound/Slot-bound/ billable); subprocess_runner.py generalizes the gw#591 boot-smoke pattern. P3 absorbs PR #353 (pgw#606/th#938) and PR #307 (pgw#583) as real hub-double dispatches (no mocking of gen_worker internals, unlike the original unit tests) - revert-verified red against lifecycle.py's pre-fix state, restored clean. P1/P2/P4/P5/P6/P7 absorb assertions from test_worker_grpc_e2e.py, test_shared_cas_root_multiwriter.py, and test_boot_smoke_gw591.py. Found + fixed in the harness itself: WorkerHarness must set the TENSORHUB_CACHE_DIR env var (gen_worker.models.cache_paths reads the process-wide get_settings() singleton, not the per-worker Settings instance) - passing tensorhub_cache_dir= to load_settings() alone silently no-ops, and every hub-double test defaults to a fresh per-call temp dir so downloaded bytes never leak between tests. 40 passed, 1 skipped (pgw#605 heartbeat: proto fields not generated yet) in ~33s local. Existing test files not touched or deleted (Phase 3). * th#960/pgw#609 Phase 2: P8 (convert publish contract) + P9 (result upload metrics), plus ruff cleanup + a hub_double(file_base_url=) knob P9 needs P8 (tests/convert/test_p8_convert_publish_contract.py, hermetic fake_hub): dtype-mismatch cast honesty (pgw#589/th#901, condensed from test_publish_as_is_dtype.py), classifier corpus table (pgw#593), and an xfail(strict=True) test-first pin for pgw#566 (open bug: normalize_adapter _state_dict only passes unet_config to converters with a NAMED parameter - diffusers' real SDXL lora_state_dict takes **kwargs only, so the SGM block remap silently never runs). pgw#569's W8A8 verifier ULP gate is a documented skip - no factored-out comparator to test hermetically without a real w8a8 artifact round trip. P9 (tests/test_p9_result_upload_metrics.py): inline vs blob_ref by size alone over a real hub-double + real local media-upload dedup sink (no S3 part scripting, matching test_media_upload_owner.py's pattern); JobMetrics typed usage survives both wire forms. Needed hub_double(file_base_url=) so the sink URL rides the FIRST HelloAck instead of a racy post-connect mutate. ruff clean (unused imports, one E731 lambda->named helper). mypy: src/gen_worker clean (untouched); new tests/harness + test files carry the same untyped- fixture-param convention as the rest of tests/ (mypy only gates src/gen_worker per CI, tests/ was never in scope). Full new suite (P1-P9 + P8's own file): 50 passed, 1 skipped (pgw#605, proto fields not generated), 1 xfailed (pgw#566, fix open) in ~31s local. * th#960/pgw#609 Phase 3: delete 4 test files fully superseded by P3/P4/P8 test_slot_boot_precedence_th938.py, test_model_slot_identity_gate.py -> superseded by test_p3_slot_binding_precedence.py (same assertions, over a real hub-double boundary instead of monkeypatched Executor.store internals). test_shared_cas_root_multiwriter.py -> superseded by test_p4_cas_multiwriter_integrity.py (same 3 tests reproduced verbatim + 1 new row). tests/convert/test_publish_as_is_dtype.py -> superseded by tests/convert/test_p8_convert_publish_contract.py (same 3 core tests against the same fake_hub). Deletions only, no other changes. Full deletion manifest (all 166 files audited, reasoning for what's NOT deleted) is in the th#960/pgw#609 tracker checkpoint - the bulk of the pre-existing suite tests production surface (CLI, LoRA runtime, quantization lanes, VRAM/RAM admission, media/upload, Slot/discovery SDK, tests/convert producer suite) that no P-test covers, so it stays pending explicit direction on the larger absorb-or-accept-loss effort that would be needed to delete it safely. * pgw#610: proto — measured per-tier DiskUsageReport on StateDelta (lockstep with th#962) * pgw#610: measured per-tier disk telemetry in every StateDelta (th#962 lockstep) - statvfs on the real mounts (CAS root = container tier; ismount-guarded fill source volume when attached; NFS tier reserved) — never a declared size - safely-reclaimable = ref-index bytes of DISK-tier refs inactive AND not in the desired set (the disk-GC LRU's eligible set); reuses ref-index bytes, no tree rescans - capacity_generation bumps only on a measured (64 MiB-quantized) shape change; rides Hello.state and every StateDelta; 30s periodic refresher is edge-suppressed * pgw#609/th#960 Phase 4: rewire CI to one suite, one lane Manual dispatch + PR-into-master only (drafts/doc-only paths run nothing); drop the on-push trigger. Same gates (mypy gating, ruff non-gating per debt note, HTTP-timeout guard, llama-server, uv build); pytest targets the whole tests/ dir so P-suite additions land automatically. * Revert "pgw#609/th#960 Phase 4: rewire CI to one suite, one lane" This reverts commit c39efb69b82d967393bd3f3243ce328d954380f7. * th#960/pgw#609 Phase 2b: absorb ~26 incident-pinned rows into P1/P2/P3/P5/ P6/P8/P9 + 2 new consolidated files, ahead of the bounded aggressive deletion sweep Bucket-level triage per the coordinator's ruling (Paul's directive: minimal greenfield set, ~92% removal). Absorbed only distinct bug-classes carrying a real pgw#/gw# incident number or a revert-catches-it story; untagged "tested thoroughly once" coverage goes without absorption (git history is the archive). P1 (+6): auth-rejection exit/retry-within-window, permanent-precondition fast-exit, hello-ack-deadline reconnect, not-leader redirect, hub-restart reconnect - absorbed from test_worker_grpc_e2e.py (#372) ahead of its deletion. P2 (+3): setup-failure FnUnavailable+recovery, host-RAM-failure wire ordering (th#807) from test_worker_grpc_e2e.py; corrupt-load quarantine+ retry-once (gw#408, J17 flood) from test_snapshot_corruption.py. P3 (+1): bare-ref prefetch binding resolution (#377) from test_executor_prefetch_binding.py. P5 (+1): heavy-dep stub contract (pgw#506) from test_discovery_heavy_deps.py. P6 (+1): SendQueue capacity-generation fencing from test_worker_grpc_e2e.py. P8 (+5): diffusers root-allinone skip (e2e J7 ENOSPC), standalone-component canonical-weight selection (gw#426), transformers sharded-index onnx/pickle exclusion, gguf explicit-quant refusal, non-safetensors-format refusal family - from test_classifier.py ahead of its deletion. P9 (+1): token-bound-owner routing (J19 run34) from test_media_upload_owner.py. New consolidated files (coordinator-authorized, one per lane with no P-test home): tests/test_quantization_lanes.py (W8A8/W4A4/FP8 contract + gw#409 promote-device-integrity, from test_w8a8.py/test_w4a4.py/ test_fp8_and_emergency_loading.py/test_promote_device_integrity.py) and tests/convert/test_convert_producer_contract.py (clone concurrency gw#442, disk preflight gw#462, download-skip bank th#592, publish resilience gw#462, from the tests/convert/ producer suite). Also reverted a premature local-only Phase 4 CI commit (c39efb6, from a prior session before an interruption) - never pushed, out of THIS issue's authorized scope ("do not touch CI yet"); preserved in history via revert, not discarded. Full new-suite gates: mypy clean, ruff clean, each new/changed file green in isolation. Deletion commit (the ~125 files these rows supersede) follows as a separate, deletions-only commit per the coordinator's mechanics. * th#960/pgw#609 Phase 3b: aggressive deletion sweep — 125 files (32,618 LoC) superseded or absorbed per the coordinator's ruling Every file here is either fully superseded by test_quantization_lanes.py / test_convert_producer_contract.py, absorbed as rows into P1/P2/P3/P5/P6/P8/ P9 (chaos b8274a1), or carries no incident pin (git history is the archive, per the coordinator's bound). Full manifest, bucket-by-bucket reasoning, and the dangling-import sweep are in the th#960/pgw#609 tracker Checkpoint 6, pushed before this commit. Deletions only, no other changes. Foreign-WIP clusters untouched: compile_cache/executor_adopt/fleet_cells (this issue's own claim-checkpoint note) and test_disk_telemetry_pgw610.py (landed on chaos by another lane during this session). Final state: 14 test files (3,293 LoC) + tests/harness/ (6 files) + conftest.py x2 + fake_hub.py, target was ~12-15 files per the coordinator. * pgw#610: lint — typed enum cast + drop unused import * pgw#610: CONTRACT.md — document StateDelta.disk_usage / DiskUsageReport * pgw#610: guard drain's disk-report cancel with getattr (stubbed Lifecycles skip __init__, same convention as _cancel_residency_reconcile) * gw#608 residual: AOT-cache disable must be process-global — torch 2.13 config user overrides are thread-local ContextVars Live disproof (0.40.5, B200, 2026-07-21): mint capture still packed 8 ASLR-keyed aotautograd entries and the store-served sibling still failed (warmups=1, cache_hits=0, cache_misses=8) — the 0.40.4 disable ran on the arming thread; the warmup compile thread saw the default True. The env var is read once at torch import (env_name_force), so post-import it is inert. Fix: entrypoint sets TORCHINDUCTOR_AUTOGRAD_CACHE=0 before any torch import (binds all threads + compile-worker subprocesses); _disable_aot_autograd_cache additionally mutates the installed config entry's env_value_force (process- global) for already-imported-torch embedders. Cross-thread revert-turns-red test added (fails on the 0.40.5 shape, passes with the fix). Dry evidence: fxgraph keys 8/8 identical across ALL five banked packs (0.40.1 x3 / 0.40.3 / 0.40.5); aotautograd level-2 keys 0/8 everywhere. samples/selfmint-proof/gw0405-run/. * th#964: family lane policy twin — bare conv-UNet (sd1/sd2/sdxl) bindings AUTO-pick scale-free #fp8 locally gen_worker.models.ladder gains family_root (modelfamily.Root twin) and the conv-UNet w8a8 exclusion table; a maybe_rebind_family_fp8 fold (hub-parity AUTO pick over the resolve's sibling_flavors, sm_89+ only, bf16 stays the sub-floor default; pins untouched) runs in the local CLI resolve ahead of the GGUF fold over one shared resolve. WorkerResolvedRepo carries the resolve's model_family (slot-declared family as fallback); select_gguf no longer lets an AUTO-ineligible w8a8 row suppress the GGUF pick. Shared vectors: none in this repo; no format change. * gw#613/th#965: universal app-level heartbeat — layer 2 of the 3-layer liveness contract (0.41.0) Hello declares heartbeat_interval_ms=30000; the pgw#610 disk-report task is promoted to the beat: it force-re-sends the full StateDelta every 30s from the asyncio event loop (the control loop that owes progress, never a detached thread), starts at startup() entry so boot hangs are covered, and keeps beating through drain until the stream closes. Edge-triggered deltas still flow as before; the beat is the same message re-sent unchanged. Contract §3 rewritten to the 3-layer model (lockstep tensorhub c03a531d). Tests: hello cadence, force-send bypasses edge suppression, beats flow while a startup coroutine is parked (the gw#612 shape), drain keeps beating until close. * gw#613 retune: 10s beat x 6 misses; disk stats measured every 30s riding every beat HEARTBEAT_INTERVAL_MS 30000 -> 10000 (hub reaps at 6 misses, ~60s). The pgw#610 statvfs/ref-index scan keeps its 30s measurement cadence behind a TTL cache so the 10s beat never turns it into a hot loop; the report still rides every StateDelta. Contract §3 event-loop discipline clause added — already true in 0.41.0: setup/warmup/compile/residency/GC all run off-loop (executor._to_thread_complete / asyncio.to_thread). * gw#612: gate self-mint publish on full capture coverage; finalize phase; run-26 root cause corrected (0.42.0) Run 26's filed post-seal_publish hang is disproven: the worker completed setup and advertised readiness; the wedge was the hub's singular compile fence treating the 2-lane record's same-identity self-attested targets as ambiguous (tensorhub lockstep fix). Real worker defect fixed: a shared mint capture missing an unexercised mandatory sibling's graphs was still published as the family cell, bricking every adopting boot (gw#611 qwen hits=1/misses=1). finalize_self_mint packs only; the executor publishes iff every capture sharer proved in, else withholds typed+loud. New 'finalize' activity phase covers the post-proof tail to readiness. * gw#613: soften run-26 narrative — app-silent, not proven-hung (gw#612 root cause: healthy-idle, hub fence starvation) * gw#613: marco_polo_wedge — event-loop-blocking liveness probe for th#965 layer-2 drills * gw#611: credit AOT-layer hits in the adopt warmup proof; calls= in the fail-closed detail; cross-process portability repro Measured mechanism for th#954's release-bricking 0/0: bundled AOT cache serves a fresh-process adopt with fxgraph counters fully silent, so a HEALTHY serving cell read cache_hits=0/cache_misses=0 and fail-closed. inductor_counters now surfaces aot_cache_hit/aot_cache_miss and the guard wrapper counts AOT hits as serving evidence (AOT stays pinned off per gw#608; a config regression now degrades to a proven boot, not a brick). Fail-closed detail carries calls= to split orphaned-wrapper (calls=0) from counter-blind serving (calls>0). New tests: real mint->pack->fresh- subprocess adopt asserts >=1 FX hit; bundled-AOT 0/0 mechanism pin; guard credits AOT hits (red on revert). Hub lockstep: tensorhub chaos 9e7dca8e quarantines cells failing their own adopt proof. * th#767: gen_worker.families.wan — WanDefaults (animegen lane) Second family after SdxlDefaults: steps/guidance/guidance_2/max_guidance for the Wan 2.2 A14B MoE envelope. Registered under tensorhub's canonical architecture root "wan22" (not a Compile(family=) shape-grid string) so repo-metadata PUT validates against the matching tensorhub schema (companion wan22.schema.json on tensorhub chaos). Lets the wan-2.2 endpoint's T2V function migrate to the Slot model and give AnimeGen-T2V its own distilled recipe (8 steps, guidance=1.0, guidance_2=1.0) via repo metadata instead of branching on checkpoint identity. * gw#614: synthesized media-modality warmup coverage (union family cells publish) + on_hello_ack model-set-diff cancel (A) _run_synthesized_warmup coverage pass: compile-target objects still unexercised after the planned jobs get media VARIANTS of the same base payloads (warmup.media_variants: one optional image/audio field filled with a generated asset, nothing else drifts) — input-routed sibling lanes (qwen edit) mint into the shared capture, so the gw#612 gate publishes the union cell instead of withholding forever; adopts prove the sibling lane instead of arming unproven. (B) th#961 defense in depth: a HelloAck whose semantic model set (resolutions+disk_refs+snapshots+hot) matches the running reconcile's target no longer cancels it (in-flight self_mint_compile survives benign plan rewrites); changed sets cancel as before. * gw#615: fix event-loop-blocking disk telemetry — the 0.40.7 LTX post-seal_publish hang pgw#610/th#962's disk_usage_report() ran statvfs()/stat() synchronously inline in Lifecycle._state_delta(), which many hot paths call directly with no event-loop offload — including right after a self-mint's seal_publish. The provider-attached VOLUME fill-source is a network-backed mount that can stall for minutes under load (exactly what a self-mint's weight download + cell pack produce). A stalled statvfs() there freezes the whole event loop: every StateDelta, RunJob dispatch, and even the th#965 heartbeat (same loop) stops until the syscall returns. ModelStore.disk_usage_report() now only reads a cache; the actual measurement (_measure_disk_usage_report) runs exclusively via refresh_disk_usage_report(), off-loop through asyncio.to_thread. Lifecycle kicks that refresh as a fire-and-forget background task (never awaited inline), gated to the existing ~30s TTL, so a stalled mount delays only its own telemetry freshness — never any send, dispatch, or heartbeat. Root-caused via git diff v0.40.5..v0.40.7 (executor.py/lifecycle.py) plus gw#612's hub-log forensics for the qwen case: gw#612 itself was hub-side (dispatch-fence starve, corrected root cause, no worker hang) and its worker-side companion fix addresses a different defect (poisoned family-cell publish on the NEXT boot). This is a distinct, genuinely-new gen-worker regression introduced by a70e7c9 (wave-2 telemetry), reproduced locally with no GPU via a stalled-statvfs monkeypatch. Tests: tests/test_disk_telemetry_pgw610.py (new test_disk_usage_report_never_measures_only_reads_the_cache, test_state_delta_never_blocks_event_loop_on_a_stalled_mount; existing tests updated to the cache/refresh split) + tests/test_heartbeat_gw613.py (new test_beats_flow_while_disk_measurement_is_stalled; existing TTL test updated to the fire-and-forget refresh). All revert-turns-red verified. Full suite 458 passed/5 skipped; ruff+mypy clean. * gw#608: FX-key forensics — a store-served proof failure names the diverging FxGraphHashDetails component; compile_seconds + per-object proof counts in the fail-closed detail * gw#608: clamp fx-divergence detail under the 2000-char activity error cap * gw#608: exhaustive fx_cache_failure_report — counts + same-key re-save sibling diff + load probes discriminate B1/B2 in one boot * gw#608 FIX: self-mint arm is transactional over the process-global cache env — no-target siblings decline before any dir mutation, arm failure restores the prior env, and a delivered-cell-seeded process never opens a capture (the store-served 8/8-miss root cause) * gw#585: consume tensorhub v4 private-input manifests — v3->v4 hard cut (PR #320 port onto chaos) Proto adopted byte-for-byte from tensorhub chaos 07e5520b (protocol 4, RunJob.input_assets=15 — 14 taken by th#913 lane — InputAsset fields 1-6; CLI NDJSON stays v1); pb2 regenerated with grpcio-tools 1.82.1. Worker resolves the exact ordered manifest via one bounded resolve call under the attempt-scoped capability, verifies size/BLAKE3/MIME/kind/decode, preserves opaque refs, sets only local_path, attempt-owned temp cleanup; endpoint build rejects Asset-bearing set/frozenset and non-string maps; deterministic traversal (declaration order, list index, sorted string map keys). Full local suite 558 passed; mypy/ruff/ timeout-lint clean. uv.lock: mechanical 0.42.0 catch-up to pyproject. * gw#585: marco-polo-attach — private-input echo probe (blake3/size/local_path, optional in-flight delay) for the v4 e2e lane * gw#585: GEN_WORKER_INTERNAL_OBJECT_HOSTS — exact-host allowlist for resolver-minted private-input URLs (datacenter-internal object stores); caller public transports keep the full SSRF gate * gw#585: marco-polo-attach reads via os.fspath (Asset is os.PathLike; no .path() method) * gw#608: fix publish-event race in the self-mint adopt test fake — publish() fills the published dict before signaling the event (CI KeyError: 'bytes') * pgw#617: hierarchical slot bindings — load base composition, substitute components from ModelBinding.components - proto: ModelBinding.components (field 5, map component -> canonical ref; mirrors tensorhub th#980); pb regenerated - ModelRef.component_overrides (tensorhub-only, sorted pairs) — part of the binding identity, so a component-only rebind derives a new instance/ residency identity; flat bindings byte-identical - executor: _effective_spec stamps overrides per dispatch; setup materializes each override ref from its own snapshot and injects the loaded module via the gw#479 components= from_pretrained mechanism; pins/held refs/digests/compile-cell binding facts include override refs - loading.load_component_override: class from the BASE model_index.json, weights from the override tree's subtree/root - typed ComponentSubstitutionError (unknown component, non-CAS ref, non-worker-loaded slot) - registry: selected_by= slots may omit default_checkpoint (th#980 relaxation mirror; unblocks ie#524 sweep of wan-2.2/sdxl/z-image) - tests: hub-double flat->substitute->flat identity proof, typed unknown- component refusal, registry relaxation (564 green) * pgw#617: mypy — dedicated slot_comps var in _effective_spec * gw#619: report a typed hardware-unsuitable fact to the hub before the boot-time CUDA probe's silent exit entrypoint.py's boot sequence (gw#529 cuda_probe) logged GEN_WORKER_CUDA_PROBE_FAILED and exit(1)'d with zero orchestrator contact — invisible to every layer of telemetry (th#986: ~20 pod deaths whose cause, CUDA-13 images on old-driver hosts per th#988, was unattributable). On a probe failure the worker now dials the hub and sends ONE HardwareUnsuitable WorkerMessage in place of Hello (cozy.scheduler wire addition, tensorhub th#988 companion commit) before exiting — bounded best-effort (~7s worst case: 2 attempts x 3s + 1s backoff), never a reason to delay the exit materially, and a silent fallback when the hub is unreachable or predates this field (old-hub FAILED_PRECONDITION rejection). cuda_probe.py: classify_probe_failure() — typed reason_class vocabulary (torch_unavailable | cuda_unavailable | driver_too_old | cuda_error | unknown) derived from CudaProbeResult.reason; driver_too_old matches torch's own "driver too old (found version ...)" message verbatim (the exact th#591/th#979/th#988 signature). hardware_report.py (new): builds the report (nvidia-smi/NVML for driver_version+gpu_name — reads the driver directly, independent of the broken CUDA runtime being reported on; torch_version/torch_cuda_version; gen_worker_version; image_digest/instance_id from Settings; worker_id/ release_id from Settings or JWT claims) and dials it over a one-shot grpc.aio Connect stream. entrypoint.py: calls report_hardware_unsuitable() right where the probe failure was previously silently fatal; logs a cuda_probe_hardware_report startup phase either way. Tests: tests/test_hardware_report.py (classify_probe_failure vocabulary; build_hardware_report degrades safely without nvidia-smi and reads driver identity when torch.cuda is down; report_hardware_unsuitable delivers to a real hub-double socket, resolves identity from JWT claims, and returns promptly-false against both an old-hub protocol rejection and a connection-refused unreachable hub). tests/test_boot_smoke_hardware_report.py (true end-to-end: a real `python -m gen_worker.entrypoint` subprocess whose probe fails on THIS box's actual driver/torch mismatch dials a real hub socket, and the hub-unreachable fallback still exits without hanging). tests/harness/hardware_report_hub.py: shared real-socket test servicers (recording_hub, old_hub, closed_port_addr). Full suite green (578 passed) plus the extended gw#591 boot-smoke contract (tests/test_boot_smoke_gw591.py, tests/test_p7_boot_smoke.py) confirmed unaffected. * ie#522: non-inference functions must never report loading_functions() _warmup_plan already excludes non-"inference" kinds from boot warmup (spec.kind != "inference"), but available_functions()/loading_functions() didn't share that gate: a declared-but-never-dispatched conversion function (a release can bundle many; only a few get invoked) sat in loading_functions() forever, since nothing was ever going to warm it. The hub's th#965 layer-3 stall watchdog takes loading_functions() at face value and kills a worker after 10 minutes of "no open activity" on any name in that list — including a name nothing was ever loading. Reproduced live: a wan-2.2 conversion release bundling z-image-w8a8-quantization alongside clone-huggingface got its pod killed at 10m15s three times running, on three different hosts, each time mid a legitimate multi-GB clone-huggingface transfer. Fix: conversion/training/dataset-kind functions are always available (matching the existing spec.slots per-dispatch-serveability carve-out just above) and never counted as loading. * ie#522: watchdog evidence blind to I/O-bound download+load phases The activity watchdog's default liveness evidence was process CPU seconds only. Two real activities are honestly alive while burning near-zero CPU: an I/O-bound network model fill (large composites, tens of GB, CPU-light by design), and the subsequent on-disk load into GPU (safetensors mmap + device placement — third-party diffusers/torch code with no progress hook). Both read as "stalled" to the hub's th#965 layer-3 rule after 10 minutes of no heartbeat, even mid genuine progress. Reproduced live twice: identical ~9.5min self_mint_compile CancelledError at phase=load, two different RunPod hosts. Fixed three ways: - _process_cpu_evidence now sums live (not just reaped) child-process CPU via psutil — covers inductor's async_compile subprocess workers, which resource.RUSAGE_CHILDREN can't see while they're still running. - note_progress() lets an I/O callback (model-download byte ticks) heartbeat the current activity directly, independent of the CPU-sampling watchdog thread; wired into cozy_snapshot.py's _on_bytes network-byte path. - _process_io_evidence adds process disk I/O bytes (psutil io_counters) to the default combined evidence — a universal, non-invasive signal for the on-disk load phase that needs no app-level hook into third-party loading code at all. Revert-turns-red verified: reverting activity.py/cozy_snapshot.py makes exactly the 3 new evidence tests fail (zero-CPU download, zero-CPU disk load, live-child-CPU), while the pre-existing dead- activity-still-stalls test keeps the detector honest. Full suite green (584 passed, 5 skipped, 1 xfailed). * gw#621: progress registry — named monotonic counters on the 10s beat + self-diagnosis New gen_worker.progress: phases register named monotonic counters (download:<ref> bytes w/ snapshot-known totals, evidence:<kind> riding the watchdog, warmup:jobs, infer:steps, upload:bytes); family prefix maps to a per-phase self-diagnosis window (code constants). activity.on_beat rides lifecycle's 10s heartbeat: one counter-carrying ActivityUpdate per beat — frozen counters included (the hub kills on counter non-advancement) — and self_stalled=true + stalled_for_ms when even the freshest counter is past its window (kill on confession, never CPU inference). Additive ActivityUpdate fields 10-16 (counter, unit, done, total, rate, self_stalled, stalled_for_ms; th#994 carries the proto source). Counters are activity- owned (finished at activity end). Downloads now carry bytes_total on the FIRST DOWNLOADING event when the snapshot knows sizes — no more 'started total=0' then silence. Also completes ie#522's 7640551, which swept the in-flight activity.py half of this work into its commit — gen_worker.progress ships here. * gw#621: tests — registry windows/self-diagnosis, counter beats, hub-double e2e confession Revert-turns-red for the ie#522 false-positive class: a CPU-quiet byte-advancing download is visibly alive on the wire (counter beats, never self_stalled) over a REAL Worker + hub double; a frozen counter still confesses, and local behavior is unchanged (the hub owns the kill). * gw#621: progress_beat re-checks _done — a beat racing completion never re-opens a terminal activity hub-side * ie#522: arm_compile() no-ops (never raises) with no active arming scope Endpoint setup() declares intent by calling arm_compile() unconditionally (sdxl, wan-2.2, ...); whether that intent is active is the release's decision — an eager registration (no compile=Compile(...) declared) already opens ArmingScope(None, ...) as a documented no-op, but arm_compile() itself still raised RuntimeError when called with no active scope. Reproduced live: wan-2.2's eager smoke (dropped compile= to skip a 51.5min self-mint compile grid per Paul's ruling) crashed at warmup/phase=load on this exact raise. Fix: no active scope -> log at info, return False (never armed), same as every other "stays eager" path. Revert-turns-red verified (4 tests updated/added in test_arm_compile_pgw517.py). Full suite green (594 passed, 5 skipped, 1 xfailed). * pgw(th#1004): @variant_of handler annotation — variant pairing emitted into the discovery manifest (variant_of/variant), dangling/self/chain targets fail the build * ie#522: WanDefaults gains an optional shift field (scheduler flow-match shift) Live finding #3: the AnimeGen card's own 8-step recipe pins FlowMatchEulerDiscreteScheduler(shift=3.0) alongside its distillation LoRA -- a step-distilled lineage needs a non-default scheduler shift, which nothing in the wan22 inference-defaults vocabulary carried. shift: Optional[float] = None -- every non-distilled repo's metadata and the resolved-defaults fallback leave the pipeline's own scheduler config untouched; only a repo that explicitly publishes shift gets one applied at the endpoint's resolution site (wan-2.2 main.py, separate commit). Mirrors tensorhub-chaos's wan22.schema.json shift addition.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes tensorhub th#938 (P0): th#912's boot-setup watcher ran
ensure_setupon the unmodified class-table spec for Slot functions. On the sdxl fleet template (pipeline=Civitai code default + tensorhub vae) the watcher waited only on the tensorhub-sourced vae, then materialized the image-baked Civitai default ->civitai_not_found-> both fnssetup_failed-> 3/3 pods retiredall_declared_functions_disabled-> release-broken (live J31, stack th919live).Precedence contract: the hub-stamped release binding is the ONLY setup source when hub-connected — delivered via Hot DesiredInstance (th#487) or dispatch-time RunJob, both rebinding through
_effective_spec. The code Slot default is the hub-less bootstrap fallback only.Change: delete the watcher's Slot branch entirely (subsumes #352's mixed-spec band-aid — an all-tensorhub Slot fn boot-setup from the code default is equally wrong when the hub stamps a different checkpoint). The gw#591 watcher remains for plain fixed tensorhub bindings, where declared == stamped by the gw#583 identity gate.
Tests: tests/test_slot_boot_precedence_th938.py — (1) live-repro (mixed defaults, vae lands, must not fetch 827184 or poison fns; fails pre-fix), (2) hub-delivery precedence (HelloAck disk+hot from the live log shape -> setup exactly once with the stamped binding), (3) all-tensorhub Slot fn never boot-setups. tests/test_slot_boot_setup_th912.py deleted (contract reversed). Full suite 1581 passed locally (1 pre-existing load flake in test_ram_admission's 2s wait_for, passes standalone); mypy/ruff clean.