Problem
run_with_watchdog in the RP2040 deployer deliberately abandons a timed-out worker thread:
|
/// Run `work` on a dedicated thread and give up after `budget`. A storport |
|
/// retry storm behind a sick hub can block the NEW.UF2 `write_all` for |
|
/// minutes with no output; this turns that hang into an actionable failure |
|
/// that feeds the fresh-enumeration retry gate. On timeout the worker thread |
|
/// is deliberately abandoned, not joined: it only touches its own locals and |
|
/// its channel send fails silently once this receiver is dropped, so the |
|
/// wedged kernel write can unblock (or not) without holding up the deploy. |
|
fn run_with_watchdog<T: Send + 'static>( |
|
budget: Duration, |
|
label: &str, |
|
work: impl FnOnce() -> Result<T> + Send + 'static, |
|
) -> Result<T> { |
|
let (sender, receiver) = std::sync::mpsc::channel(); |
|
std::thread::spawn(move || { |
|
let _ = sender.send(work()); |
|
}); |
|
match receiver.recv_timeout(budget) { |
That worker owns the open NEW.UF2 destination file. If the kernel write remains blocked, the thread and its file handle stay alive inside the daemon after fbuild reports a timeout. This can contaminate retries and later deploy attempts against the same volume — an open exclusive handle on NEW.UF2 can make every subsequent attempt fail for a reason unrelated to the device.
Identified in #1161 as a recovery hazard introduced by the #1083 hardening (debc9963). This bug stands even after the PICOBOOT transport flip (sibling sub-issue), because the mass-storage copy remains the fallback transport.
Current test gap
The existing watchdog test only verifies the caller returns quickly by leaving a five-second worker running. It does not verify worker termination, handle release, or whether an immediate second write can open the destination.
Proposal
- Rework the timeout path so an abandoned write cannot hold the destination hostage. Options, roughly in preference order:
- Open/write in a way the watchdog can cancel (e.g.
CancelSynchronousIo on the worker thread handle on Windows, then join with a bounded grace period).
- If the write genuinely cannot be cancelled, at minimum ensure retries use a distinct destination strategy or detect-and-report the stuck-handle state explicitly rather than failing opaquely.
- Track abandoned workers in the daemon (count + age) and surface them in deploy diagnostics so a wedged handle is visible instead of silent.
Acceptance
- New test: after a watchdog timeout on a deliberately-blocked write, the destination path is openable for exclusive write within the grace period (or the stuck state is explicitly reported).
- New test: daemon diagnostics report abandoned worker count.
- No change to the happy-path write shape.
Part of the meta issue tracking the #1161 transport strategy change. Defaults chosen by the drafting agent are listed inline — edit as needed.
Problem
run_with_watchdogin the RP2040 deployer deliberately abandons a timed-out worker thread:fbuild/crates/fbuild-deploy/src/rp2040.rs
Lines 544 to 560 in 16bb9ff
That worker owns the open
NEW.UF2destination file. If the kernel write remains blocked, the thread and its file handle stay alive inside the daemon after fbuild reports a timeout. This can contaminate retries and later deploy attempts against the same volume — an open exclusive handle onNEW.UF2can make every subsequent attempt fail for a reason unrelated to the device.Identified in #1161 as a recovery hazard introduced by the #1083 hardening (
debc9963). This bug stands even after the PICOBOOT transport flip (sibling sub-issue), because the mass-storage copy remains the fallback transport.Current test gap
The existing watchdog test only verifies the caller returns quickly by leaving a five-second worker running. It does not verify worker termination, handle release, or whether an immediate second write can open the destination.
Proposal
CancelSynchronousIoon the worker thread handle on Windows, then join with a bounded grace period).Acceptance
Part of the meta issue tracking the #1161 transport strategy change. Defaults chosen by the drafting agent are listed inline — edit as needed.