feat(dash-spv): rewindable filter scan for mid-session wallet import

[lklimek]

Problem

When a wallet is imported into a running SPV client that has already synced past the wallet's first-relevant block height, historical UTXOs belonging to that wallet become permanently invisible to the SPV layer until the SPV client is fully torn down and restarted — and in some cases not even then, because a fourth constraint in the persistent-storage layer re-anchors the scan floor back at the tip after reload. There is no public API today to trigger a bounded rescan after a wallet's script set expands.

Downstream effect for dash-evo-tool: mid-session mnemonic imports discover zero balance even when on-chain UTXOs exist. Users must quit and re-launch the app for historical transactions to appear — and depending on the on-disk segment state, even that is not reliably sufficient. See the consumer issue dashpay/dash-evo-tool#829 and PR #830 for the full context — DET is implementing a programmatic SPV stop-and-restart as a workaround, which is heavy-handed and discards unrelated sync progress.

Root cause — four compounding constraints

1. Birth height is hardcoded to current sync tip on import

key-wallet-manager/src/lib.rs:316 in WalletManager::import_wallet_from_extended_priv_key:

let mut managed_info = T::from_wallet(&wallet);
managed_info.set_birth_height(self.synced_height);   // ← current sync tip

Same pattern repeats at lines 357, 405, 450. This means any wallet imported mid-session starts with birth_height = synced_height, i.e. already past anything historical. The add_wallet (line 134) and one import_wallet_from_bytes variant (~line 210) accept an explicit birth_height parameter — but even passing 0 alone is not sufficient because of (2), (3), and (4).

2. FiltersManager reads scan floor only at construction

dash-spv/src/sync/filters/manager.rs:161–165:

(wallet.earliest_required_height().await, wallet.filter_committed_height())

These values are captured once when FiltersManager::new() runs. After that, the manager's internal processing_height, active_batches, and progress.committed_height() evolve based on its own state. Changing a wallet's birth_height post-construction does not propagate. No public method re-evaluates the floor.

3. filter_committed_height is monotonic; rescan_batch only covers active batches

WalletInterface::update_filter_committed_height (wallet_interface.rs:103–107) is strictly forward-only:

fn update_filter_committed_height(&mut self, height: CoreBlockHeight) {
    if height > self.synced_height() {
        self.update_synced_height(height);
    }
}

There is no public setter to lower it. FiltersManager::rescan_batch is pub(super) and triggered exclusively by BlockProcessed events carrying new_addresses from maintain_gap_limit during in-pipeline block processing — it cannot re-scan committed batches (they have been removed from active_batches via self.active_batches.remove(&batch_start) at line 499 of filters/manager.rs).

4. SegmentCache::first_valid_offset anchors get_start_height() near the tip after reload

dash-spv/src/storage/segments.rs:128–134 in SegmentCache::load_or_new:

if let Some(segment_id) = min_seg_id {
    let segment = cache.get_segment(&segment_id).await?;
    cache.start_height = segment
        .first_valid_offset()
        .map(|offset| Self::segment_id_to_start_height(segment_id) + offset);
}

Even if a caller fully tears down the SPV client, resets filter_committed_height to 0, and reconstructs FiltersManager fresh against the expanded wallet set, the freshly-loaded DiskStorageManager computes start_height from the first non-sentinel offset in the min-id segment — and in practice that value returns near the chain tip, not the actual segment floor, anchoring FiltersManager's scan_start away from historical filters.

Observed on Dash testnet (dash-evo-tool log trace 2026-04-20T12:40:11Z):

• Previous SPV run initialized from checkpoint height 1,400,000, then ran parallel header sync from 1400000 to 1461916 and Filter sync complete at height 1461917 — segments 28 (heights 1,400,000–1,449,999) and 29 (heights 1,450,000–1,499,999) on disk each hold tens of thousands of written headers/filters.
• Wallet is imported. DET tears down the SPV runtime, sets wm.update_filter_committed_height(0), and rebuilds a new SpvClient against the same data directory.
• New DiskStorageManager loads, FiltersManager::new() reads wallet.earliest_required_height() = 0 and wallet.filter_committed_height() = 0 — correct.
• But header_storage.get_start_height() returns 1,461,917 (= segment_id_to_start_height(28) + first_valid_offset(segment 28) where first_valid_offset returns 61,917) instead of 1,400,000.
• FiltersManager::start_download computes scan_start = wallet_birth_height.max(header_start_height) = 0.max(1_461_917) = 1_461_917.
• Log line: Loading stored filters 1461917 to 1461917 into current batch — only a single filter at the tip is re-scanned. Historical filters (1,400,000–1,461,916), where the imported wallet's UTXOs live, are never re-matched.

Either first_valid_offset under-reports valid entries on reload, or the written-offset tracking isn't durably persisted across runs. Whichever the precise fault, the practical outcome is that any proposed fix (Option A/B/C/D below) must either bypass this floor (e.g. let callers force a scan floor below get_start_height()) or fix the underlying offset computation so the scan floor reflects where headers actually begin on disk.

Reference scenario

A testnet wallet with a multi-output transaction confirming 10 DASH outputs at BIP44 external indices 0 and 32–40 (gap of 32, well beyond the default gap limit of 30). The motivating tx: 56bdab0ac5dfefc47e136eedeede07a83ff0f2cb753683578fd41677975f8b32 in block 1,459,352 on Dash testnet. Importing this wallet into an already-synced SPV client never surfaces the UTXOs at 32–40, even after the consumer (dash-evo-tool) pre-extends the wallet's BIP44 monitored set to cover indices 32–40 before announcing the wallet to SPV. The filter batch containing block 1,459,352 was already committed; no rescan path reaches it.

Proposed API additions (any one of these would be sufficient; ideally a combination)

Option A — Rewindable filter_committed_height + dynamic scan-floor re-read

Minimum: Allow WalletInterface::update_filter_committed_height(h) to accept values lower than current synced_height for the specific purpose of "re-scan required from here." FiltersManager listens to a signal (event, revision counter, explicit hook) and rewinds its processing_height / committed_height / active_batches to match, then continues scanning against the current monitored set.

Option B — Public rescan_from_height(h) on DashSpvClient / FiltersManager

A single entry point callers invoke after expanding a wallet's script set:

/// Request that the filter scan be re-run from `height` upward for any
/// wallets whose `earliest_required_height` has changed. Persisted
/// filters/headers are reused; no network re-download unless missing.
pub async fn rescan_from_height(&self, height: CoreBlockHeight) -> Result<(), Error>;

Option C — WalletManager::register_addresses(wallet_id, &[Address]) with rescan signal

A public bulk register method that does NOT require repeated get_receive_address(mark_used=true) calls (which have the side effect of advancing highest_used cosmetically in the wallet's internal tracker). Combined with a rescan signal, the caller can:

1. Expand a wallet's monitored set atomically.
2. Trigger bounded rescan covering the new scripts.

Option D — Dynamic re-evaluation of WalletInterface::earliest_required_height

Make FiltersManager re-query earliest_required_height on each tick (or in response to a monitor_revision bump) and rewind its state when the floor drops below processing_height.

Why this is worth doing upstream

• DET's current workaround is full SPV restart — and even that is not reliably sufficient because of constraint (4): the freshly-loaded storage re-anchors the scan floor at the tip, so a restart against an existing data directory can still fail to surface historical UTXOs. The only bullet-proof workaround available today is wiping the SPV data directory and re-downloading from the checkpoint, which is user-visible (minutes of re-sync) and discards all in-flight sync progress.
• PR fix: separate filter scan height from synced height #442 author's own TODO — the PR that introduced filter_committed_height explicitly flagged the design as provisional: "It's not the best solution I think, hence I added the TODO for now. I will look into this at some point later." This issue formalizes the follow-up.
• The infrastructure is mostly there. PersistentBlockStorage (feat: introduce PersistentBlockStorage #397), persistent filter storage, the maintain_gap_limit → new_addresses → BlockProcessed → rescan_batch wiring from feat: capture new addresses from maintain_gap_limit #287 and feat: rewrite, fix and improve the sync architecture #411 all exist. What's missing is a public entry point to say "re-scan range X..Y against the current wallet set", a rewindable committed-height invariant, and a SegmentCache scan floor that honors what's actually persisted.

Out of scope for this ticket

• Fixing import_wallet_from_extended_priv_key to accept an explicit birth_height is a separate, smaller change. It's necessary but not sufficient — see constraints (2), (3), and (4).
• The out-of-order block processing issue (bug: out-of-order block processing causes SPV wallet to miss UTXO spends #649) is unrelated to this proposal.

Related

• dashpay/dash-evo-tool#829 — consumer bug
• dashpay/dash-evo-tool#830 — DET-side atomic pool extension + restart workaround
• dashpay/rust-dashcore#442 — filter_committed_height introduction (author's TODO)
• dashpay/rust-dashcore#287 — new_addresses from maintain_gap_limit
• dashpay/rust-dashcore#411 — sync architecture rewrite (current rescan_batch mechanism)
• dashpay/rust-dashcore#634 — single-map WalletManager refactor (foundation for per-wallet state)

_{🤖 Co-authored by Claudius the Magnificent AI Agent}