fix(app): never freeze on tray quit during a backup; quitting tray state; honest recovery status

[pmaxhogan]

Closes #299
Closes #300
Closes #301

PR1 of the v2.12.0 wave.

Diagnosis

#299 - tray quit freezes the app (and trips WER) during a backup

Root cause: the graceful drain ran on the platform event-loop thread.

tray.rs menu_id::QUIT -> app.exit(0) -> RunEvent::ExitRequested ->
shutdown_orchestrators(), which drove the whole drain inline via
tauri::async_runtime::block_on. block_on blocks the calling thread, and
the RunEvent callback runs on the main thread that owns the Win32 message
pump. So the pump stalls for the entire drain budget: RUN_LOOP_DRAIN_TIMEOUT
is 20s and graceful orchestrator shutdown is only observed between cycles, so
quitting mid-backup reliably parks there. Windows ghosts a window that has not
pumped for ~5s, paints "Not Responding", and Windows Error Reporting kills the
process.

Evidence, from the owner's machine (maxbook, Windows 11 26200):

• Event Log, Application Hang provider:
  • 8/16/2026 10:49:23 AM - "The program driven-app.exe version 2.11.2.0
    stopped interacting with Windows and was closed."
  • plus two more for 2.10.1.0 / 2.10.0.0 on 8/14. AppHangB1 is the
    blocked-message-pump signature, not a crash.
• driven-diagnostics-hung-on-starting-backup.zip,
  logs/driven.2026-08-16.log:

  15:49:09.170Z  INFO driven::app: explicit quit; draining orchestrators
  15:49:09.170Z  INFO driven::app: signalling graceful shutdown on quit account_id=...
  15:49:29.415Z  INFO driven::logging: rolling file logs active      <- a NEW process
  

  The hang event lands 14s into that gap (10:49:23 local == 15:49:23Z). The
  quitting process never wrote all per-account tasks shut down (no orphans) -
  it was killed mid-drain. Same shape on the second quit: 15:52:59.576Z quit
  -> new process at 15:53:12.858Z.

So the freeze did not even buy the graceful semantics it was blocking for: the
drain is killed before it finishes and the run loop dies hard anyway.

#301 - ~1 minute of opaque "Starting backup"

Two independent defects, both visible in the same log:

15:53:34.330Z  cycle start tick=Manual
15:53:34.330Z  state transition state="power_check"
15:53:34.735Z  reconcile: recovering pending ops pending_ops=18
15:54:39.929Z  state transition state="scanning"      <- 65.2s later

1. Serial round trips. reconcile_inner walks pending ops strictly one at
   a time, and every op costs at least one Drive metadata round trip
   (metadata by id for the update path, find_by_op_uuid under a re-derived
   parent for the create path). 65.2s / 18 ops == ~3.6s per op, which is a
   normal Drive files.list latency. An earlier run in the same log shows the
   same 60.0s shape.
2. No status at all. OrchestratorState::Recovering was emitted only
   from the byte-level resume branch, via the on_recover sink. The 18
   plain adopt-or-requeue ops move no bytes, so they emitted nothing and the
   orchestrator stayed on PowerCheck for the whole pass - which is the
   generic "Starting backup..." the user saw. (In the 15:49 run, where one op
   did carry a resumable session, Recovering did appear - for 2 of the 32
   seconds - which is why the bug looked intermittent.)

What changed

#299 quit never blocks the event loop

• lib.rs: shutdown_orchestrators is split into take_shutdown_handles
  (synchronous, cheap - every call is a mutex-take or a watch-send, no await,
  no I/O) and async fn drain_shutdown_handles (the unchanged drain body).
• begin_graceful_quit hides the window and repaints the tray synchronously
  (single fast platform calls - the window disappearing is the whole point of
  clicking Quit), then spawns the drain on the Tauri async runtime and
  returns. The event loop keeps pumping for the entire drain, so the app can
  never be declared hung.
• RunEvent::ExitRequested becomes a three-phase machine on a process-global
  QUIT_PHASE:
  • IDLE -> prevent_exit(), start the drain, return at once;
  • DRAINING -> a second explicit quit (repeat --quit, OS session end) is
    prevented and ignored; "Force quit now" is the deliberate escape hatch;
  • READY -> not prevented, so the process exits. The drain sets READY and
    calls app.exit(0) itself when it is genuinely done.
• The cancellation-safety invariants documented at the old lib.rs:228-238 and
  app_state.rs:209-215 are preserved verbatim: still no outer
  tokio::time::timeout around the cancellation-UNSAFE
  AccountHandle::shutdown sweep, still join_all for concurrency, still
  drain_or_abort's self-bounding await-then-abort-and-await per handle. The
  VSS/APFS broker shutdowns still run after the orchestrator drain, now at
  the tail of the spawned task.

#300 quitting tray state

• New TrayIcon::Quitting: slate (#71717a) badge with a white stop
  square glyph. The approved mockup used amber; amber is already spent on
  Paused and NetworkAttention, and "quitting" must not read as "paused,
  still running". Slate is the only desaturated badge in the set, and the stop
  square is what carries the state on the macOS template path where hue is
  discarded entirely.
• Tooltip tray.tooltip.quitting: "Driven - quitting (finishing backup,
  right-click to force quit)".
• tray::enter_quitting swaps the menu for a two-item one: a disabled
  status line "Quitting - finishing current backup..." and a single enabled
  "Force quit now". Every normal action is deliberately gone - they would all
  queue work against orchestrators that are already winding down.
• crate::force_quit marks the phase READY and exits immediately. This is
  safe to do hard: the streaming resumable uploader persists
  resumable.acked_offset into the pending op's payload after every acked
  wire chunk (executor.rs push_one), so the next launch's reconcile resumes
  from the last acked byte and at most one wire chunk is re-sent. Restore jobs
  stage into a temp that is never promoted, so a kill cannot publish a partial
  file.
• apply_state returns before touching the tray while quitting (the run loop
  is still finishing its cycle and keeps emitting transitions, which would
  otherwise repaint the spinner over the quitting icon), and skips the OS
  notification too - a "first sync complete" toast while the app is closing is
  noise. The aggregate per-account map is still updated.
• The macOS menu bar title engine clears its title once and stops ticking while
  quitting, so a live "12.3 MB/s, 4 min left" cannot contradict the icon.

#301 faster recovery + honest status

• RECONCILE_LOOKUP_CONCURRENCY = 6. prefetch_reconcile_lookups runs the
  adopt-or-requeue lookups through buffer_unordered ahead of the unchanged
  sequential decision pass.
  • Independence: remote.metadata and find_by_op_uuid are pure reads; the
    create path's parent walk (ensure_parents_once) is idempotent AND already
    single-flighted behind parent_walk with a per-source parent_dirs cache,
    built precisely so concurrent uploads into one new directory cannot race
    duplicate ensure_folder calls. Nothing in the prefetch touches the
    state DB.
  • Durability is unchanged: every delete_pending_op / adopt_reconciled /
    clear_file_state_drive_file_id still runs one at a time, in the original
    op order, in the sequential pass - no whole-batch buffering (the 2026-08-14
    OOM incident was reconcile buffering too much, so this deliberately buffers
    only lookup results).
  • Pacing is unchanged: each lookup still takes pacer.permit_request() and
    reports its ResponseClass, so a rate limit or an open circuit breaker
    still gates the pass. The accounting simply moved into the two lookup
    helpers alongside the calls.
  • Fail-fast: the sequential pass aborts the source's reconcile on the first
    retryable lookup error, so prefetching could turn one failed request
    during a Drive outage into N. The first future to see one sets a halt flag;
    every not-yet-started future returns None, and the pass makes that one
    lookup inline and hits the same abort.
  • Excluded from the prefetch: ops carrying a live resumable session (they
    stream a whole file and own the byte ticks, so they stay strictly
    sequential; their fall-through lookup when a session is stale is done
    inline) and ops with no client_op_uuid.
• OrchestratorState::Recovering gains ops_done / ops_total, fed by a new
  OP flavour of RecoverProgress emitted once before the first round trip and
  once per recovered op. reconcile_inner stamps the pass's live counters onto
  every tick, byte ticks included, so the deep resume call sites did not
  have to thread them down. The orchestrator's existing ~1/s throttle now
  treats "final" as both dimensions being complete.
• UI: progress.recoveringOps, a determinate percent from the op counters when
  no bytes are moving, and the label "Recovering - resuming 7 of 18 uploads".

Test results

All run locally on the Mac unless noted.

• cargo test -p driven-core --lib - 559 passed, 0 failed
• cargo test -p driven-app - 442 + 6 + 3 passed, 0 failed
• cargo clippy --workspace --all-targets -- -D warnings - clean
• cargo fmt --all --check - clean
• pnpm -C ui run test:unit - 793 passed (59 files)
• pnpm -C ui run build (vue-tsc + vite) - clean
• pnpm -C ui run lint - 0 errors (35 pre-existing unused-i18n-key warnings)
• pnpm -C ui run format:check - clean
• On real Windows (maxbook, x86_64-pc-windows-msvc, worktree at
  V:\driven-agents\wave1): cargo check -p driven-app --lib and
  cargo clippy -p driven-app -p driven-core --all-targets -- -D warnings both
  exit 0. (Cross-compiling from macOS is not possible - ring/aws-lc-sys
  need a Windows C toolchain - so this was checked on the machine itself.
  Note: building this repo on maxbook now needs NASM for aws-lc-sys 0.43,
  pulled in by the dependabot bumps on main; I installed it via
  winget install NASM.NASM.)

New tests:

• executor::reconcile_lookups_run_concurrently_and_report_op_progress - 12
  ops against a fake store with a fixed 120ms per-request delay. A serial pass
  has a hard floor of 1440ms; the assertion is < 720ms, and two concurrent
  rounds land near 240ms. This fails if the prefetch is removed or silently
  serialised - it is not a sleep-and-hope timing test. Also asserts the first
  tick announces 0 of 12 before any lookup, every tick carries the total, the
  final tick reports 12 of 12, and ops_done is monotonic.
• executor::reconcile_with_no_pending_ops_emits_no_op_ticks - a clean boot
  stays free (no transition churn).
• executor::reconcile_lookup_classifies_which_failures_halt_the_prefetch -
  transient metadata/orphan failures and a parent-walk failure halt; a
  definitive not-found does not (it is per-op and the pass continues).
• executor::resume_emits_recover_progress_ticks updated for the new tick
  stream (byte ticks are now those with a non-zero total; op counters asserted
  on every tick).
• tray: TrayIcon::Quitting added to every exhaustive icon test - distinct
  colour, distinct glyph, badge actually painted, static across frames, brand
  dimensions, and the macOS template STATES set.
• i18n: tray.quitting_status, tray.force_quit, tray.tooltip.quitting
  added to both the exact-label and the no-raw-key sweeps.
• UI: op-counter percent, byte-over-op precedence, cross-account summing, and
  the new progress-bar label.

README

Updated in this PR (repo rule):

• the resumable-upload bullet now covers the op-count readout and the
  concurrent startup recovery;
• a new bullet documents that quit never freezes, the quitting tray icon, and
  "Force quit now".

Remaining QA on maxbook (for the team lead, after a dev build exists)

The unit + Windows-toolchain checks above cannot exercise the message pump, so
these need the installed/dev app on maxbook:

1. [bug] Windows: app freezes (WER dialog) when quitting from tray during an active backup #299 primary repro. Start a backup with real work in flight (the
   existing source has multi-GB files), then Quit from the tray. Expect: the
   window disappears immediately, the app stays responsive, the process lives
   until the drain finishes, and the log ends with
   all per-account tasks shut down (no orphans) ->
   graceful quit drain complete; exiting. Then confirm no new
   Application Hang event:
   Get-WinEvent -FilterHashtable @{LogName='Application'; ProviderName='Application Hang'} -MaxEvents 5
2. feat(app): tray shows a distinct 'quitting' state with a Force quit now action #300 tray affordance. During that drain, check the tray icon changed to
   the slate stop badge, hover for the tooltip, and right-click: the menu must
   be exactly the disabled "Quitting - finishing current backup..." plus
   "Force quit now".
3. feat(app): tray shows a distinct 'quitting' state with a Force quit now action #300 force quit. Repeat, then click "Force quit now" mid-drain. Expect
   an immediate exit, and on the next launch the log shows
   reconcile: resuming persisted resumable upload (streaming re-read) with a
   resume_from at or near where it was killed (not 0).
4. [bug] backend sits on 'Starting backup' for ~1 min before flipping to resume #301 timing. With a backlog of pending ops (the diag bundle showed
   18-22), time from reconcile: recovering pending ops to the first
   state transition state="scanning". Baseline was 65.2s; expect roughly a
   sixth of the round-trip time. During it, the app should read
   "Recovering - resuming N of M uploads" with a moving determinate bar, never
   a generic "Starting backup...".
5. Sanity: quit with nothing running still exits promptly.

Not merged - the team lead babysits the merge.

---

Follow-up commit: coverage

The first push failed the coverage gate at Rust 84.43% vs main 84.56%
(epsilon 0.1pp) - the new prefetch error branches had no tests behind them.
be55fe9 fixes that:

• ReconcileLookup drops its ParentFailed variant in favour of a nested
  Orphan(Result<Result<..>>) (outer = the parent walk, inner =
  find_by_op_uuid). That removes the one defensive-but-unreachable arm the
  first shape needed - the create branch is selected by the absence of a
  recorded drive_file_id, so the "not prefetched" fallback covers it.
• reconcile_parent_walk_failure_aborts_the_pass_and_keeps_every_op - 8 nested
  create ops against a store whose ensure_folder always fails. Covers the
  parent-walk propagation, the halt classifier's Orphan(Err) arm, and (with
  more ops than the concurrency bound) the prefetch's halt branch, and asserts
  every op is kept.
• reconcile_adopts_an_update_whose_object_already_carries_the_op_uuid - the
  UPDATE path's successful metadata lookup, which had no test at all before.

All 17 checks green on be55fe9; mergeStateStatus: CLEAN. Windows
cargo test + clippy passes, so the concurrency timing assertion is not flaky
on the slowest runner.

[github-actions]

Coverage

Area	main	this PR	delta
Rust (lib crates)	84.58%	84.50%	-0.08 (OK)
UI (vue/ts)	93.05%	93.06%	+0.01 (OK)

Gate: passed - no coverage regression (epsilon 0.1 pp).