node: 'Block not found' 404s from upstream block-fetcher on Mutinynet (~17/h on dfxdev)

[TaprootFreak]

Symptom

On dfxdev, zkcoins-node emits ~17 error lines per hour, all of the form:

Error fetching block txids 0000026bb85eab422e4694aa2d19146d95b2f63a46ef2b152209fd2a603da54d:
  HttpResponse { status: 404, message: "Block not found" }

In the last 60 min sample: 17 lines, 17 unique block hashes — i.e. each error is a distinct block, not the same one retried.

Hash pattern (0000026b..., 000002b9..., 000000c6...) is consistent with Mutinynet / Bitcoin signet difficulty (small leading-zero prefix, well below mainnet's 19+ zeros).

Likely cause

Two candidates, both plausible — needs verification:

1. Mutinynet reorgs. Signet has frequent reorgs by design. The node observes a block hash via one source, requests txids from the txindex/electrs upstream, and by then the block has been re-orged out → 404. If true, this is expected steady-state noise, not a bug.
2. Upstream pruning / race. The node sees a block tip and races the upstream's indexing — the block exists on the node but is not yet exposed via the txid endpoint.

To tell which: check whether the 17 hashes per hour cluster around the chain tip (recent height) or are spread across older heights. Tip-clustered → reorg. Spread → indexing race.

Why this isn't (necessarily) a bug

If the cause is reorgs, the right fix is not to silence the error but to:

• Reclassify the log line to warn for the 404-on-stale-block case (per feedback_log_noise_reclassify.md — downgrade, don't drop)
• Keep error if the same hash 404s repeatedly (real upstream failure, not a transient reorg)

A heuristic: track each block hash in a small LRU; if a hash is requested once and 404s once and is never seen again → it was reorged out, log warn. If the same hash 404s N times → log error.

Acceptance

• Decide: reorg vs indexing race (check height distribution of the 404'd hashes against getbestblockhash at the time of the error)
• If reorg: split the log line into two paths — warn for first-seen-and-gone, error for repeated 404s on the same hash
• If indexing race: add a short retry-with-backoff (e.g. 3× over 5 s) before logging at error level
• Verify on dfxdev: error-rate should drop to < 2/h

Verification query

sum(count_over_time(
  {server="dfxdev", container_name="zkcoins-node"}
  |= "Block not found" [60m]
))

Target: < 2/h after fix.

Context

• Surfaced 2026-06-01 ~11:00 CEST during a sweep of all error-level logs on dfxprd/dfxdev (Loki, line-content regex (?i)(^|\s)error[\s:\[]).
• Only seen on dfxdev (Mutinynet). PRD (Mainnet) zkcoins-node was not in the top-15 error producers.
• Total scope: one of several uncovered noise sources; sibling issues track dEURO/JuiceDollar Apollo timeouts and LDS boltz/LND noise in DFXServer/server (private infra repo).

cc — opening here because zk-coins/node has Issues enabled and zkcoins owns this code outright.