feat(reference-index): bounded server-side wait on IndexBehind to avoid client retry storms

[panos-xyz]

Background

The morph_getTransactionHashesByReference RPC currently enforces a strict global tip gate: it only serves a query when the durable index cursor exactly matches the canonical head (indexed_to == tip.number && indexed_hash == tip.hash, see crates/reference-index/src/reader.rs::query_at and the before/after chain_info() checks in crates/rpc/src/morph/handler.rs). Otherwise it returns -32000 "reference index is behind" (IndexBehind).

Because the index runtime is asynchronous (PR #141 deliberately decoupled it from block execution to protect sync throughput), there is a short race window once per block: the new block becomes the canonical tip (provider.chain_info() reflects it immediately) before the runtime finishes indexing it. Any query landing in that window gets IndexBehind, even queries that do not touch the newest block.

Problem

Two user-facing symptoms:

1. "Submit a MorphTx, then immediately query its reference" — a very common, high-frequency pattern. The reference lives in the newest block, which the runtime may not have indexed yet, so the client sees IndexBehind and must retry (poll-until-ready, similar to waiting for a receipt).
2. The failure rate is sensitive to block time. At second-level block production the window is ~1% of wall-clock; if millisecond-level block production lands (feat(consensus): mirror block time switch to milliseconds (go-ethereum#332) #123) the same few-ms indexing window becomes a much larger fraction, making retries noticeably more frequent.

geth does not have this state at all: its reference index is written atomically with the block (WriteReferenceIndexEntriesForBlock in the same batch as body/receipts/tx-lookup), so the index never lags the head and the RPC simply reads whatever is in the DB.

Proposed fix (Option 1: bounded server-side wait)

Instead of returning IndexBehind immediately when the cursor lags the tip, have the RPC handler wait briefly for the runtime to catch up before deciding:

• On lag, poll the ReferenceIndexHandle cursor/phase until indexed_to == tip (and hash matches) or a small timeout elapses (e.g. 50–100 ms).
• The steady-state lag is only a few milliseconds, so the vast majority of waits resolve within one or two RECONCILE_POLL_INTERVAL-scale ticks and return real data.
• Only if the wait times out do we return IndexBehind (genuinely behind / rapid-sync Deferred / repairing).

This converts the "client keeps retrying" experience into a single transparent server-side wait, without changing the RPC response shape or wire semantics.

Details to settle in the PR

• Timeout value + whether it is configurable (default ~50–100 ms).
• The handler method is blocking (synchronous MDBX reads); the wait must not tie up the blocking pool longer than the timeout — bound it strictly and re-check chain_info() after the wait (the tip may itself have advanced).
• Interaction with RapidSyncDetector Deferred: when the runtime has intentionally stopped indexing (rapid sync), don't wait the full timeout pointlessly — detect the phase and fail fast.
• Metrics: count waits, wait duration, and timeouts to observe how often this fires in production.

Alternatives considered

• Option 2 (relax to geth semantics): drop the global tip gate; return all matches within the indexed range plus an indexed_to marker. Closer to geth, but changes the response structure and requires downstream cooperation. Tracked as a possible follow-up if Option 1 proves insufficient.

Scope

• crates/rpc/src/morph/handler.rs (wait loop) and possibly a small helper on ReferenceIndexHandle in crates/reference-index/.
• No change to the storage layer or the runtime write path.
• Decoupled from refactor(reference-index): replace ExEx with canonical cursor runtime #141 and perf(reference-index): bound reorg rewind floor to the L1 finalized block #143 — separate PR.