Mint state-desync class: num_pubkeys advances past on-chain commitments after absorbed publisher errors

[TaprootFreak]

TL;DR

The server's minting account counter (minting_meta.num_pubkeys) can drift past the SMT/MMR contents of on-chain commitments. Once this happens, every subsequent /api/mint and /api/send for the minting account returns 422 because account_server::get_merkle_proofs cannot find the previous commitment in state. The only currently-known recovery is a full DEV-state wipe.

Two distinct manifestations of this class were observed today:

Time (UTC)	Failure point	Error
2026-05-23 ~10:30	state.get_commitment_proof(public_key)	Unable to get merkle proofs for provided public key
2026-05-23 ~17:10	state.get_mmr_inclusion_proof(previous_root)	Unable to get mmr inclusion proof for the previous root

Both fail with HTTP 422 to the client (via the api-e2e mint_roundtrip_lands_balance_and_proof and send_commit_roundtrip_moves_balance tests against live DEV).

Root cause

DEV_SKIP_BROADCAST_FAILURE=true (set on dfxdev's compose) tells mint_handler / send_handler to swallow a publisher-broadcast failure, log it, and continue serving the client. The handler still:

1. Advances minting_meta.num_pubkeys in Postgres.
2. Calls account_server.receive_coin, which updates the in-memory SMT.
3. Returns 200 OK to the client with a proof_id.

But the on-chain commit/reveal transactions were never broadcast, so the scanner never ingests them. The next mint or send for the same minting-account-pubkey-N hits get_merkle_proofs (or get_mmr_inclusion_proof, depending on which derivation path runs first) for the previous pubkey, that proof isn't in state, and the request 422s.

This is documented in infrastructure/dfxdev/zkcoins/docker-compose.yaml next to the env var:

"On-chain commitment is missing on the failed broadcast; the next state change that depends on the SMT having the previous public key will fail until /data is wiped."

So the failure mode is known but the design accepts it as a recovery-by-wipe.

Reproduction (deterministic)

1. Ensure the Mutinynet publisher wallet (currently tb1pvyhjfd90wmlw2dyjjdg2k0zqfp6mr9sufu4ztjwu78z2t0h5pjgshwz8up) has zero UTXOs.
2. Trigger any mint against DEV (e.g. via api_remote::mint_roundtrip_lands_balance_and_proof or the wallet app).
3. Mint returns 200 OK. Publisher-broadcast logs No UTXOs available for inscription broadcast. State advances.
4. Top up the publisher wallet.
5. Trigger another mint — fails with Unable to get merkle proofs for provided public key (or MMR variant) → 422.

Reproduces 100 % of the time when step 1 holds.

Why current mitigations don't fully solve it

PR / commit	Scope	Why it doesn't close this issue
#81	Postgres migration checksum revert	Unrelated — only fixes sqlx VersionMismatch on container start
#82	reset_state=true workflow_dispatch refactor	Refactor only — used to be inert, now calls host-side dispatcher
DFXServer/server#229	Host-side dispatcher also DROP SCHEMA	Makes the recovery-by-wipe complete; doesn't prevent the bug
#83	api_remote::send_commit_roundtrip retry on scanner-lag 422	Test-side mitigation for one specific 422-message pattern. The 17:10 failure today returned a different message (`mmr inclusion proof`) that the retry doesn't match.

Proposed paths forward (need decision)

Option A — remove DEV_SKIP_BROADCAST_FAILURE now that the publisher is funded

• Publisher currently has ~98 884 sats (one UTXO from inscription cb9c873e…4242).
• With the dispatcher fix from DFXServer/server#229, reset is one workflow-dispatch click away if publisher ever empties again.
• Pro: removes the silent-success path entirely — clients get 503 instead of state-drift.
• Con: api-e2e roundtrips fail loud whenever publisher is briefly empty (rare with monitoring).

Option B — keep the flag, but don't advance state on broadcast failure

• The handler treats broadcast failure as a soft-rollback: log + return 503 + don't bump num_pubkeys and don't receive_coin.
• Pro: keeps the flag's intent (tolerate flaky DEV broadcasts) while preventing the desync.
• Con: requires careful unwinding of any in-handler state mutation (the MintProof-into-ProofStore step also runs before broadcast; needs review).

Option C — add a startup state-consistency check

• On connect_and_migrate + load_state, assert mmr.size() >= minting_meta.num_pubkeys. Refuse to start if violated.
• Pro: a regression of either above option surfaces at boot, not at request-time.
• Con: doesn't fix the underlying race — just catches it earlier.

A + C in combination is probably the right answer pre-mainnet. B alone is incremental.

Risk assessment

• DEV only today. PRD has DEV_SKIP_BROADCAST_FAILURE unset — the corresponding code-path returns 503 on broadcast failure and never advances state. PRD does not have this bug today.
• PR-A1..A3 enlarged the blast radius: pre-A3 the desync was just three file-blobs that could be deleted; post-A3 the state spans Postgres tables that until DFXServer/server#229 (today) couldn't be wiped via the reset_state workflow.
• Pre-mainnet blocker? Once mainnet is configured, the lack of `DEV_SKIP_BROADCAST_FAILURE` means PRD itself is safe — but anyone running a self-hosted server with the flag accidentally enabled would inherit the same bug. Worth a comment in docker-compose.yaml.template and README.md to forbid it outside DEV.

Related

• Issue #84 (event-driven scanner) reduces the scanner-lag failure window but doesn't fix the absorbed-broadcast case.
• The publisher-wallet topup happened today at TX 97e34f29b83298b3976b295a8d976d740a0ba08e25132971d93bd5a5130dd91d (100 000 sats).