Skip to content

th#1031: cell_selection_bug recovers via self-mint instead of blocking every request#371

Merged
PaulFidika merged 1 commit into
masterfrom
release/th1031-cell-selection-bug-recovery
Jul 23, 2026
Merged

th#1031: cell_selection_bug recovers via self-mint instead of blocking every request#371
PaulFidika merged 1 commit into
masterfrom
release/th1031-cell-selection-bug-recovery

Conversation

@PaulFidika

Copy link
Copy Markdown
Contributor

Root cause (confirmed against live pod_events)

Two stacked compile-cache failures, both live on pod_events.class='cell_selection_bug':

  1. Signature mismatch is real and persistent, not caused by pgw#622. 54 rows across 2 releases (7f57379eeab6464e96c9234c, ec4335c9700c445c09bb0c3c) and 9 pods over 24h+. The earlier release's rows start 2026-07-22 08:16 UTC -- ~12h before pgw#622 was even committed (20:10 UTC same day) -- so pgw#622 is coincidental timing, not the cause. Every row rejects the same stale published signature c3c4395265ad against a consistently-different consumer signature (826a1212a266, mostly). cell_key (gw#581/th#883) has no graph-shape axis, so a stale/incompatible sdxl graph can sit published under a key that a structurally-different (or since-changed) consumer legitimately never matches.

  2. cache_collision on the hub-pushed re-adopt is a downstream symptom, not a separate bug: because CellSelectionBugError aborted arming outright (fleet_cells.enable_compiled re-raised it verbatim, th#883's original "never mask it" invariant), the worker could never self-mint a corrected replacement. The hub kept re-pushing the same stale cell; each retry cycle left a fresh but non-matching local capture that later collided on adopt.

Net effect: every request on the affected worker paid a full self_mint_compile phase=warmup_forward cycle, because the only path back to a compiled state (self-mint) was structurally unreachable from the cell_selection_bug exit.

Fix

ArmOutcome now carries the caught CellSelectionBugError. fleet_cells.enable_compiled still treats it as a bug (unchanged th#883 wire visibility -- callers still send the cell_selection_bug ModelEvent/pod_event) but no longer aborts arming: it falls through to self-mint exactly like an ordinary miss, so the worker recovers into a working, published-and-replacing compiled state instead of retrying the identical unusable cell forever. Self-mint's own fail-closed safety net (no CUDA/toolchain/target) is untouched -- a genuine mint impossibility on a mandatory lane still refuses, chained from the selection bug via __cause__ so the report is never dropped.

Threaded through both compile-arming seams: the worker-loaded slot path (_enable_compiled) and the self-loaded pipeline path (ArmingScope/arm_compile -- what sdxl actually uses) via a new selection_bugs map on the scope.

Tests

  • tests/test_fleet_cells.py: rewrote the invariant test (test_cell_selection_bug_propagates_untouched -> test_cell_selection_bug_reports_and_self_mints) to lock in report-then-recover; added test_cell_selection_bug_still_fail_closed_when_mint_impossible for the genuine-impossibility fail-closed path.
  • cc.enable()/cell_key tests (the lower-level primitive that still correctly raises) untouched -- this fix is purely in the fleet orchestration layer's handling of that exception.
  • Full suite green locally: 636 passed, 5 skipped, 1 xfailed.

Scope

SDXL-family only in the observed pod_events (no other family shows this class). Architecturally latent fleet-wide wherever cell_key doesn't fully capture graph-shape compatibility -- worth a follow-up to add a graph-signature axis to cell_key itself (bigger blast radius: invalidates every published cell fleet-wide, so deliberately out of scope for this hotfix). Filed as a follow-up note in th#1031.

Full writeup: ~/cozy/cozy-creator-tracker/tensorhub/progress.md th#1031.

…locking forever

A self-requested cell whose graph signature drifts from this runtime's own
(cell_key has no graph-shape axis, so structurally different sdxl graphs can
share one key) used to raise CellSelectionBugError straight out of
fleet_cells.enable_compiled, aborting the whole arm attempt. For a mandatory
w8a8/w4a4 lane that becomes CompiledLaneUnavailableError, which fails setup
entirely and gets retried from scratch on the next request/job -- hitting the
identical stale delivered cell again, forever. Live pod_events proves this:
54 cell_selection_bug rows across 2 releases/9 pods over 24h+ (predating
pgw#622 by ~12h, so unrelated to it), all rejecting one stale signature
(c3c4395265ad) that nothing was ever able to overwrite.

ArmOutcome now carries the caught CellSelectionBugError; enable_compiled
falls through to self-mint (the ordinary MISS recovery) instead of raising.
The th#883 invariant stays wire-visible -- callers still send the
cell_selection_bug ModelEvent/pod_event -- it just no longer blocks serving.
Self-mint's own fail-closed safety net (no CUDA/toolchain/target) is
untouched, so a genuine mint impossibility on a mandatory lane still refuses,
now chained from the selection bug via __cause__.

Threaded through both compile-arming seams: the worker-loaded slot path
(_enable_compiled) and the self-loaded pipeline path (ArmingScope/
arm_compile, what sdxl actually uses) via a new selection_bugs map so
neither silently drops the alarm.

(cherry-picked from chaos 97cf56b; hand-reapplied onto master's
executor.py -- chaos has unrelated indentation-changing refactors in the
same function that master doesn't have yet)
@PaulFidika
PaulFidika marked this pull request as ready for review July 23, 2026 07:49
@PaulFidika
PaulFidika merged commit 1ddf3a2 into master Jul 23, 2026
2 checks passed
@PaulFidika
PaulFidika deleted the release/th1031-cell-selection-bug-recovery branch July 23, 2026 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant