bug(dapi): stale nonce/balance read after confirmed broadcast due to DAPI replica lag

[Claudius-Maginificent]

bug(dapi): stale nonce/balance read after confirmed broadcast due to DAPI replica lag

Expected Behavior

After transfer_address_funds_with_nonce (or any address-based state transition) returns Ok — meaning wait_for_state_transition_result has confirmed the tx is included in a block — the very next SDK read of that address's nonce or balance should reflect the committed state. Any subsequent state transition built on those values should be accepted by consensus.

Current Behavior

The SDK's gRPC client round-robins across DAPI replica nodes. After a confirmed broadcast, a sibling replica may not yet have applied the block. The next fetch_inputs_with_nonce (or AddressInfo::fetch) may land on the lagging node and return a stale nonce N even though the on-chain nonce is N+1. The SDK then builds a new state transition with nonce N, and consensus rejects it:

ConsensusError(StateError(AddressInvalidNonceError(AddressInvalidNonceError {
    address: P2pkh([...]),
    provided_nonce: N,
    expected_nonce: N+1   // or N+2 if mempool-awareness skips ahead
})))

The error appears on the second address-based operation in a sequence, after the first was successfully confirmed. This is a read-after-write consistency failure, not a chain inclusion lag.

The same shape applies to balance reads: a lagging replica returns an old balance, causing AddressDoesNotExistError or Insufficient combined address balances on the immediately following state transition.

Steps to Reproduce

1. Target testnet (or any multi-node network) with --test-threads=2.
2. Call bank.fund_address (which calls transfer_address_funds_with_nonce) for two addresses concurrently.
3. Immediately after the confirmed return, issue a second address-based state transition (e.g. another fund call or register_identity_from_addresses) without a chain-confirmed-streak poll in between.
4. Observe ~50% failure rate with AddressInvalidNonceError { provided_nonce: N, expected_nonce: N+1 } (or N+2).

No special setup required beyond a live multi-node testnet. The race is timing-dependent; --test-threads=1 reduces but does not eliminate it.

Possible Solution

Workaround (in-tree): The e2e framework already documents and implements the multi-streak polling pattern. See:

• packages/rs-platform-wallet/tests/e2e/framework/wait.rs lines 73–93 — prose explaining the DAPI round-robin race (Marvin QA-802 findings).
• Lines 352–380 — wait_for_address_balance_chain_confirmed_strong, the stronger four-hit variant used when even the standard streak gate isn't enough.

The workaround is to call wait_for_address_balance_chain_confirmed_n (or the _strong variant) after every confirmed broadcast and before any dependent state transition. This is a test-framework band-aid, not a protocol fix.

Upstream fixes to consider:

Layer	Option
DAPI	Pin a gRPC session to a single replica for read-after-write consistency window
DAPI	Serve reads from a quorum (consistent but adds latency)
SDK	After wait_for_state_transition_result succeeds, poll until LSN >= confirmed block height before returning to caller
DPP / proof verifier	Include block height in gRPC response so client can detect and retry stale reads

Context

Encountered when running rs-platform-wallet e2e tests on v3.1-dev at --test-threads=2 against testnet. Without the chain-confirmed-streak helper between consecutive address operations, roughly half of the subsequent fund operations panic with the typed nonce error. The race is not a chain inclusion lag — wait_for_state_transition_result already confirms inclusion. It is specifically inconsistency across DAPI replica reads.

Related (distinct root cause): #3407 — addresses AddressInvalidNonceError arising from a missing stale-mark after a failed broadcast, not from replica lag after a confirmed one.

Your Environment

• Version: v3.1-dev branch HEAD (as of 2026-05-07)
• Environment: testnet, multi-node DAPI cluster
• Affected packages: packages/rs-sdk, packages/rs-dapi-client, packages/rs-platform-wallet (tests)

---

_{🤖 Co-authored by Claudius the Magnificent AI Agent}