Skip to content

fix(rp2040): watchdog-abandoned UF2 write worker leaks the NEW.UF2 handle inside the daemon #1164

Description

@zackees

Problem

run_with_watchdog in the RP2040 deployer deliberately abandons a timed-out worker thread:

/// Run `work` on a dedicated thread and give up after `budget`. A storport
/// retry storm behind a sick hub can block the NEW.UF2 `write_all` for
/// minutes with no output; this turns that hang into an actionable failure
/// that feeds the fresh-enumeration retry gate. On timeout the worker thread
/// is deliberately abandoned, not joined: it only touches its own locals and
/// its channel send fails silently once this receiver is dropped, so the
/// wedged kernel write can unblock (or not) without holding up the deploy.
fn run_with_watchdog<T: Send + 'static>(
budget: Duration,
label: &str,
work: impl FnOnce() -> Result<T> + Send + 'static,
) -> Result<T> {
let (sender, receiver) = std::sync::mpsc::channel();
std::thread::spawn(move || {
let _ = sender.send(work());
});
match receiver.recv_timeout(budget) {

That worker owns the open NEW.UF2 destination file. If the kernel write remains blocked, the thread and its file handle stay alive inside the daemon after fbuild reports a timeout. This can contaminate retries and later deploy attempts against the same volume — an open exclusive handle on NEW.UF2 can make every subsequent attempt fail for a reason unrelated to the device.

Identified in #1161 as a recovery hazard introduced by the #1083 hardening (debc9963). This bug stands even after the PICOBOOT transport flip (sibling sub-issue), because the mass-storage copy remains the fallback transport.

Current test gap

The existing watchdog test only verifies the caller returns quickly by leaving a five-second worker running. It does not verify worker termination, handle release, or whether an immediate second write can open the destination.

Proposal

  1. Rework the timeout path so an abandoned write cannot hold the destination hostage. Options, roughly in preference order:
    • Open/write in a way the watchdog can cancel (e.g. CancelSynchronousIo on the worker thread handle on Windows, then join with a bounded grace period).
    • If the write genuinely cannot be cancelled, at minimum ensure retries use a distinct destination strategy or detect-and-report the stuck-handle state explicitly rather than failing opaquely.
  2. Track abandoned workers in the daemon (count + age) and surface them in deploy diagnostics so a wedged handle is visible instead of silent.

Acceptance

  • New test: after a watchdog timeout on a deliberately-blocked write, the destination path is openable for exclusive write within the grace period (or the stuck state is explicitly reported).
  • New test: daemon diagnostics report abandoned worker count.
  • No change to the happy-path write shape.

Part of the meta issue tracking the #1161 transport strategy change. Defaults chosen by the drafting agent are listed inline — edit as needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions