Problem
enter-mixed-transition requires a live runtime registered with transition_role=queue_storage_target. The role exists because a 0.6 runtime binds its execution engine once at startup: an auto runtime started pre-flip resolves canonical and can only downgrade to drain-only when the flip lands — it never starts executing queue-storage work. So the gate demands an explicit witness that someone will execute the new plane; storage_capability=queue_storage alone can't prove that.
In practice this forces every embedding application to plumb TransitionWorkerRole::QueueStorageTarget through its own config/deploy machinery for a one-time operation, then remove it (or leave a permanently misleading builder call). Verified on 0.6.5 that a live queue-storage-capable auto runtime is refused, and that a fully quiesced cluster (zero fresh heartbeats) is also refused.
Proposal 1 — auto runtimes re-resolve at the flip (removes the role)
Runtimes already observe transition state changes — that's how auto runtimes downgrade to canonical_drain_only after the flip. Instead of only downgrading, an auto runtime that observes mixed_transition could re-resolve and bring up its queue-storage runtime in place (internally close to "restart the client": a runtime started fresh during mixed_transition already resolves to queue storage, so the resolution logic exists; this is supervisor-level re-init, not new engine code).
The gate then relaxes to "≥1 live runtime with storage_capability=queue_storage", and TransitionWorkerRole can be deprecated for operators. This also fixes two adjacent sharp edges observed in a staged-transition rehearsal:
- a pre-flip maintenance leader never promotes queue-storage
deferred_jobs (its promotion path branches on its startup-resolved engine), so due queue-storage jobs sit unpromoted until leadership moves to a post-flip runtime;
- the "roll all workers after finalize" step stops being load-bearing.
Proposal 2 — allow the flip on a quiesced cluster (much smaller)
If no runtime has a fresh heartbeat, allow enter-mixed-transition. With nothing live there is no fleet to strand: nothing can execute canonical work either, and any 0.6 runtime started after the flip resolves to queue storage at startup. This gives operators a role-free path: scale the fleet to zero, flip, (optionally migrate the deferred backlog, see #456), finalize, scale back up. For small/bursty deployments a brief quiesce window is often easier than coordinating a witness runtime.
Implementation wrinkle: the gate lives in the database (awa.storage_enter_mixed_transition, installed by v014), so relaxing it in SQL means a new migration on the 0.6 line — awkward with 0.7 owning v041–v043. Alternative: a CLI/awa-model-side path (e.g. enter-mixed-transition --quiesced) that performs its own liveness check and the state transition directly, refusing if any fresh heartbeat exists.
The two compose: quiesced clusters flip with no role today; live clusters keep the witness until proposal 1 lands, after which the role disappears entirely.
Problem
enter-mixed-transitionrequires a live runtime registered withtransition_role=queue_storage_target. The role exists because a 0.6 runtime binds its execution engine once at startup: an auto runtime started pre-flip resolves canonical and can only downgrade to drain-only when the flip lands — it never starts executing queue-storage work. So the gate demands an explicit witness that someone will execute the new plane;storage_capability=queue_storagealone can't prove that.In practice this forces every embedding application to plumb
TransitionWorkerRole::QueueStorageTargetthrough its own config/deploy machinery for a one-time operation, then remove it (or leave a permanently misleading builder call). Verified on 0.6.5 that a live queue-storage-capable auto runtime is refused, and that a fully quiesced cluster (zero fresh heartbeats) is also refused.Proposal 1 — auto runtimes re-resolve at the flip (removes the role)
Runtimes already observe transition state changes — that's how auto runtimes downgrade to
canonical_drain_onlyafter the flip. Instead of only downgrading, an auto runtime that observesmixed_transitioncould re-resolve and bring up its queue-storage runtime in place (internally close to "restart the client": a runtime started fresh duringmixed_transitionalready resolves to queue storage, so the resolution logic exists; this is supervisor-level re-init, not new engine code).The gate then relaxes to "≥1 live runtime with
storage_capability=queue_storage", andTransitionWorkerRolecan be deprecated for operators. This also fixes two adjacent sharp edges observed in a staged-transition rehearsal:deferred_jobs(its promotion path branches on its startup-resolved engine), so due queue-storage jobs sit unpromoted until leadership moves to a post-flip runtime;Proposal 2 — allow the flip on a quiesced cluster (much smaller)
If no runtime has a fresh heartbeat, allow
enter-mixed-transition. With nothing live there is no fleet to strand: nothing can execute canonical work either, and any 0.6 runtime started after the flip resolves to queue storage at startup. This gives operators a role-free path: scale the fleet to zero, flip, (optionally migrate the deferred backlog, see #456), finalize, scale back up. For small/bursty deployments a brief quiesce window is often easier than coordinating a witness runtime.Implementation wrinkle: the gate lives in the database (
awa.storage_enter_mixed_transition, installed by v014), so relaxing it in SQL means a new migration on the 0.6 line — awkward with 0.7 owning v041–v043. Alternative: a CLI/awa-model-side path (e.g.enter-mixed-transition --quiesced) that performs its own liveness check and the state transition directly, refusing if any fresh heartbeat exists.The two compose: quiesced clusters flip with no role today; live clusters keep the witness until proposal 1 lands, after which the role disappears entirely.