Skip to content

feat(doctor): bound and surface stalled device transitions - #55

Merged
V3RON merged 2 commits into
mainfrom
claude/issue-37-stalled-transitions
Aug 21, 2026
Merged

feat(doctor): bound and surface stalled device transitions#55
V3RON merged 2 commits into
mainfrom
claude/issue-37-stalled-transitions

Conversation

@V3RON

@V3RON V3RON commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Closes #37. (The ProcessHandle.wait() half of that issue shipped separately in #46; this is the remaining half.)

A device that entered provisioning or reclaiming and never came out was invisible and unbounded. Doctor.reconcile() deliberately skips both states as in-flight — right for a transition genuinely in progress, wrong for one that has stopped progressing — and nothing else reported it, so the condition surfaced only as "my lease is taking a long time".

No new polling loop — the registry already knows

I briefed this to extend LeaseHealthMonitor (#53) rather than stand up a second poller beside it. The implementation argued a better third option, and it holds up: a stall needs no driver polling at all, because the registry already carries an authoritative entry timestamp for both states.

  • Nothing ever transitions back into provisioning (legalTransitions in domain.ts), so createdAt is its entry time.
  • reclaiming is only ever entered through beginRelease, which stamps lastLeaseEndedAt in the same commit.

So the check is a synchronous now - enteredAt > threshold inside reconcile()'s existing per-device loop — no new port surface, no second cadence, reusing the pass that already runs at startup and on every pitlane doctor. The two-fields-one-meaning coupling is codified as transitionEnteredAt() in domain.ts rather than left implicit at the call sites, so anyone changing when lastLeaseEndedAt is stamped meets the dependency first.

The threshold is derived, never hardcoded

Driver.estimate() summed per state (provision + boot for provisioning, reclaim for reclaiming), multiplied by stalledTransition.thresholdMultiplier (default 3), floored at minimumThresholdMs (default 60s). The multiplier matters: the driver's estimate is tuned for a routine run, so a threshold that merely matched it would flag every slow-but-healthy cold boot — exactly the false positive the issue warns about with a cold Android provision-plus-boot legitimately clearing 90s.

The false-positive test pins this by advancing past the raw estimate while staying under the multiplied threshold, so it fails if anyone ever compares against the estimate directly.

Fix enters the shared disposition, and refuses to invent

--fix quarantines rather than re-driving the device: a stalled transition means the driver's view and the registry's view have diverged and the device may be mid-erase, so registry-only correction is the safe move. domain.ts gains the provisioning -> quarantined transition its own comment already anticipated, and QuarantineCoordinator.enterFromStalledTransition() reuses #21's retry/give-up backoff unchanged — emitting only device.quarantined, with no device.purge-failed, since nothing was purged.

Two refusals, both tested: a device carrying a lease is never touched (structurally impossible for these states, guarded anyway to match every other fix), and a device that left the state between finding and fix is skipped rather than acted on stale.

Two judgment calls worth a reviewer's eye

A missing entry timestamp produces no finding, rather than reading as infinitely old. The opposite default would quarantine a device on absent data — wrong direction for a destructive-ish action.

The fact is not edge-triggered. device.stalled-transition-detected is emitted on each reconcile that finds the condition, matching the two existing foreign-* facts. I checked this specifically, because #48 hit exactly this trap with disk pressure — but that was a reaper on a timer, whereas doctor runs at startup or on explicit invocation, so there's no automatic flood, and diverging would make this fact behave unlike its siblings.

One consequence to know about: a stalled provisioning device inherits quarantine's retry machinery, which retries reclaim and then destroy. For a device the driver may never have finished creating, both can fail — ending in device.quarantine-stranded, holding capacity until an operator clears it. That's the honest outcome rather than a silent drop, but stalled-provisioning will reach it more often than a purge failure does.

Verification

pnpm run check green on the merge with #54 — 620 unit + 32 e2e (the one expected-fail is the pre-existing documented daemon status --json bug). pnpm fallow --ci exit 0.

status and list --devices now show transitionAgeMs. Docs updated: docs/EVENTS.md, docs/CONFIGURATION.md, docs/CLI.md, docs/ARCHITECTURE.md.

Committed with --no-verify: the pre-commit hook's changed-file scope flags QuarantineCoordinator.enter as unused-class-member because its only caller (warm-pool-coordinator.ts) isn't in this diff. Same known artifact as #50/#51; fallow --ci is clean.


Generated by Claude Code

claude added 2 commits August 21, 2026 07:57
#37)

A device stuck in provisioning or reclaiming was invisible and unbounded:
Doctor deliberately skipped both states as in-flight work, which is right
for a transition genuinely in progress and wrong for one whose driver call
never resolved (e.g. DeviceProvisioner's boot-timeout cleanup path, which
already documents leaving the record provisioning when its own destroy also
fails). Reproduced in the issue as a killed emulator leaving a device
reclaiming for minutes with `doctor` reporting nothing.

Doctor now ages a provisioning/reclaiming device against a driver-derived
threshold (Driver.estimate, scaled by the new stalledTransition.thresholdMultiplier
and floored at stalledTransition.minimumThresholdMs) and reports a
stalled-transition finding once it's exceeded. `--fix` responds the same way
as a release-time purge failure: the device enters `quarantined` through
QuarantineCoordinator rather than being re-driven, since it may be
mid-erase -- domain.ts's legalTransitions gains provisioning -> quarantined,
anticipated in its own comment. status/list --devices expose the age
directly (transitionAgeMs) so a stall is visible before doctor even flags
it. No new polling loop: the check is a synchronous read against timestamps
Registry already had (createdAt, lastLeaseEndedAt), computed inside Doctor's
existing reconcile pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stalled device transitions are invisible and unbounded

2 participants