Wallet mutex held across `Runtime::block_on(persist_async)` can deadlock the entire node runtime

[jkczyz]

Summary

The on-chain Wallet holds its inner (and persister) std::sync::Mutex across Runtime::block_on(locked_wallet.persist_async(...)) at 11 sites in src/wallet/mod.rs (e.g. apply_update, <Wallet as Listen>::block_connected, get_change_script_inner, select_confirmed_utxos_inner, get_new_address). Other async tasks on the node runtime acquire that same mutex synchronously from worker threads (e.g. <Wallet as Listen>::block_connected during bitcoind-RPC chain sync, list_confirmed_utxos_inner / get_change_script_inner from the anchor BumpTransactionEventHandler via the async WalletSource).

When the runtime is short on worker threads, this forms a three-party deadlock that halts the entire runtime:

1. Task A locks the wallet mutex, then parks in Runtime::block_on(persist_async) (block_in_place + Handle::block_on), waiting on a KV store write future.
2. Task B, polled on a runtime worker, blocks synchronously on the wallet mutex held by A, wedging its worker core. With one worker (worker_threads = 1, as most integration tests configure — Runtime::new picks up the ambient handle) the runtime is now fully halted: no task anywhere can be polled.
3. The write future A is waiting on is queued behind a shared async primitive in the store's write path whose current holder is a task on that same runtime. That task can never be re-polled because of (2), so it never releases the primitive; A never finishes persisting; A never releases the wallet mutex; B never unblocks.

Runtime::block_on's block_in_place hands the worker core off before parking, so A itself doesn't wedge the runtime — but it can't prevent B from doing so, and it holds the wallet mutex the whole time. This is the same failure class already described on StoreRuntime in src/runtime.rs, but the coupling resource here isn't a store's I/O driver — it's any tokio::sync primitive in any KVStore write path — so the store-runtime isolation doesn't cover it.

Evidence

Observed as an indefinite hang of a WIP integration test (rbf_splice_payment_reverts_after_deep_reorg, currently part of #962; it deep-reorgs an anchor channel into a force-close) when random_chain_source lands on TestChainSource::BitcoindRpcSync. Reproduced on the first or second try by pinning the chain source; macOS sample captured two hangs that are the same cycle with the roles of legs 1 and 2 swapped:

Hang 1 (legs: event handler holds the lock, chain sync wedges the core):

• Background-processor task, off-core, parked forever: EventHandler::handle_event → BumpTransactionEventHandler::handle_channel_close → select_confirmed_utxos → WalletSource::get_change_script → Wallet::get_change_script_inner (locks inner, then persister) → Runtime::block_on → tokio::task::block_in_place → Handle::block_on(persist_async) → CachedParkThread::park.
• Chain-sync task, owning the only worker core, blocked forever: BitcoindChainSource::poll_and_update_listeners_inner → ChainNotifier::connect_blocks → <ChainListener as Listen>::block_connected → <Wallet as Listen>::block_connected → Mutex::lock on the wallet inner mutex.

Hang 2 (roles swapped):

• Chain-sync task, off-core, parked forever: <Wallet as Listen>::block_connected (locks inner, applies the block, locks persister) → Runtime::block_on(persist_async) → parked.
• Background-processor task, owning the only worker core, blocked forever: BumpTransactionEventHandler::handle_channel_close → select_confirmed_utxos → list_confirmed_utxos → Wallet::list_confirmed_utxos_inner → Mutex::lock on the wallet inner mutex.

In both samples the process contained only 4 threads: the two above plus the test-harness pair — with the single worker core wedged, the runtime had no thread left that could poll anything. (tokio 1.52 runs worker cores on blocking-pool threads and migrates them on block_in_place; previous carrier threads idle out after ~10s.)

Leg 3 evidence. In the test environment the coupling primitive is TestSyncStore's serializer: tokio::sync::RwLock<()> (tests/common/mod.rs), whose write guard is held across .awaits (spawn_blocking hops) inside every store op. Instrumenting all serializer acquisitions with a watchdog (std thread, so it keeps running while the runtime is dead; dumps ops outstanding >20s) captured a third hang, same shape as hang 1, with a stable set of exactly three stuck ops for the entire 7-minute observation window:

[SER-WATCHDOG] op702 stuck 97s: W(write) monitor_updates/b8c47e9a.../4 | holding-write-lock
[SER-WATCHDOG] op703 stuck 97s: W(write) monitors//b8c47e9a..._0
[SER-WATCHDOG] op704 stuck 97s: W(write) bdk_wallet//indexer

op702 is a channel-monitor persist task (spawned onto the node runtime via RuntimeSpawner, i.e. chainmonitor::AsyncPersister → FutureSpawner): it acquired the serializer write lock, hit its next .await, and can never be re-polled — the only worker core is wedged on the wallet mutex. op704 is the BDK wallet persist issued from inside get_change_script_inner's block_on(persist_async) — the wallet-mutex holder — queued behind it forever. (op703 is another monitor persist in the FIFO between them.) All three legs of the cycle are visible in this single capture.

Production stores are exposed to the same cycle whenever a write future awaits a primitive held by a task on the node runtime; and with an owned runtime (multiple workers) the wedge requires the wallet mutex to block that many workers at once — harder, but the ingredients (per-block Listen calls, force-close bump events, API calls all taking the wallet mutex) all rise together during reorg/force-close storms, and users can hand ldk-node a small external runtime via Builder.

Why only bitcoind sync hits it in tests

<Wallet as Listen>::block_connected is only driven by the bitcoind chain source (ChainNotifier-based sync); it both (a) takes the wallet mutex synchronously on a worker for every connected block and (b) itself persists under that mutex (src/wallet/mod.rs Listen impl). Esplora/Electrum share the lock-across-block_on pattern via apply_update, but don't have the per-block synchronous listener path, so the contention window is far smaller.

Repro recipe

1. Pin random_chain_source to TestChainSource::BitcoindRpcSync.
2. Run an integration test that force-closes an anchor channel while blocks are being connected (we hit it with the deep-reorg splice test from Fix funding-payment reclassification downgrade and deep-reorg duplication #962; the deep reorg produces a BumpTransactionEvent::ChannelClose concurrently with Listen::block_connected calls and a stream of monitor persists). It hung on the first try, twice in a row, and also hangs at earlier points in the test (any overlap of a wallet persist, a wallet-mutex acquisition on the worker, and an in-flight store write suffices).
3. sample <pid> shows the stacks above.

When this was introduced

The cycle assembled over three changes, all long on main before the test that exposed it:

• ed12e65 ("Bump to LDK main (4e32d85)", 2025-02) — the anchor BumpTransactionEventHandler starts driving the wallet through the async WalletSource (get_change_script_inner, list_confirmed_utxos_inner).
• 4879002 ("Introduce Runtime object allowing to detect outer runtime context", 2025-05) — Runtime::block_on becomes block_in_place + Handle::block_on on the ambient runtime.
• 5ac2ed1 ("Use BDK's async wallet persister") and b3f947c ("Move DataStore persistence onto async KV storage"), both 2026-06-03 — the final ingredient: wallet and payment-store persistence under the wallet mutex became async, bridged with block_on. Before this migration the wallet's critical sections were synchronous I/O that always completed on their own, so leg 3 of the cycle could not exist.

So the deadlock has been reachable since the June 2026 async-storage migration. #962 itself touches none of the involved code (verified against its head, f01e83e) — its new test merely lines the trigger up reliably, so hangs on that PR's CI (roughly 1 in 3 runs draw the bitcoind chain source) are this pre-existing bug, not the PR.

Where a fix could go

• The root invariant violation is holding the wallet inner/persister mutexes across block_on(persist_async) (leg 1). E.g. extracting the staged changeset and persisting outside the wallet lock (the persister already merges changesets internally), or making these paths natively async so they await instead of block_on. This breaks the cycle at its strongest link.
• Narrower mitigations look tempting but don't fully close it: wrapping the bitcoind listener dispatch in tokio::task::block_in_place unwedges hang 1, but hang 2's core-wedger is the async WalletSource path (list_confirmed_utxos_inner) — every synchronous wallet-mutex acquisition reachable from a runtime worker would need the same treatment.
• Raising test worker_threads merely lowers the probability; the cycle only needs all workers to be blocked on the wallet mutex at once.

Environment

ldk-node main (deadlock-relevant code identical at f04fa4d), tokio 1.52.3, LDK rev 3dfcc4cca186, macOS arm64. Full sample outputs and the watchdog patch available on request.

---

This bug was investigated and this report drafted with an AI assistant (Claude Code); the stacks, instrumentation output, and reproduction are from real local runs.

[jkczyz]

I asked Claude to one-shot fix this: jkczyz@df66130

@tnull What do you think about that approach?

[tnull]

Good catch!

I asked Claude to one-shot fix this: jkczyz@df66130

@tnull What do you think about that approach?

Hmm, holding the persister in a tokio::sync::Mutex makes sense, but it might be nice to already prepare for further async'ification, i.e., go directionally towards #926, i.e., making more internal API methods async and moving to fewer remaining block_ons that are called closer to the API boundary. Will open a draft PR that combines the two shortly.

[tnull]

Now opened #980