Non-deterministic eth_call reads silently corrupt indexing (POI divergence)

[rickstaa]

Summary

Several handlers read contract state with eth_call. graph-node runs each call against the RPC at the historical block and trusts the response, so a degraded or lagging backend yields a plausible-but-wrong value that gets persisted. try_ never aborts at the call site — it takes the reverted branch and writes a fallback — so the damage surfaces later as wrong data, as an abort in an unrelated handler, or as POI divergence between indexers running identical code.

We hit the delayed-abort variant at block 144056642 and shipped a targeted patch (#249). This issue tracks the general class.

Rule of thumb: a try_ fallback is safe only when the revert branch is bounded to a known block range or network and its value is indistinguishable in effect from the truth. Otherwise it is silently manufacturing state.

Call sites

Call site	Fallback	Blast radius	Aborts	Self-heals
try_getFirstTranscoderInPool roundsManager.ts:54	EMPTY_ADDRESS → loop skipped	no Pool for the whole round	was yes, fixed by #249	no
try_getTotalBonded roundsManager.ts:66	0	round/protocol/day stake + 5 fields skipped	no	partially
try_slot0 helpers.ts:472	price 0	7 cumulative USD accumulators	no	no
try_getGlobalTotalSupply livepeerToken.ts:23,76	totalSupply ± amount	totalSupply, participationRate	no	yes
getNextTranscoderInPool (bare) roundsManager.ts:130	n/a — reverts abort	truncated walk on a wrong-but-valid answer	yes, loudly	no

1 — getFirstTranscoderInPool. The original incident. A revert leaves currentTranscoder at EMPTY_ADDRESS, the loop at roundsManager.ts:122 never runs, and no Pool exists for the round. A revert means "the pool is empty", impossible post-launch, so the fallback is never legitimately taken.

2 — getTotalBonded. Same handler, same block, same RPC as #1 — they fail together. Still unguarded. Zero is written to round/protocol/day (roundsManager.ts:75,226,229), and the gt(ZERO_BD) guard at roundsManager.ts:232 then skips participationRate, delegatorsCount, inflation, numActiveTranscoders, activeTranscoderCount on both round and day. protocol.totalActiveStake = 0 persists until the next newRound, zeroing day.totalActiveStake on every mint/burn/winningTicketRedeemed in between.

3 — slot0. Price 0 flows via getEthPriceUsd() into seven .plus() accumulators (ticketBroker.ts:81,90,105,109,130,138,151), so it permanently under-counts USD volume. getPriceForPair also returns ZERO_BD by design off-Arbitrum, making 0 an overloaded sentinel — "wrong network", "pool not yet deployed", and "RPC hiccup" are indistinguishable. The LPT/ETH pool 0x4fd47e… has no code at the arbitrum-one start blocks, so the zero path is genuinely exercised early in a sync.

4 — getGlobalTotalSupply. Written correctly, listed for completeness. The fallback protocol.totalSupply ± amount is arithmetically right in normal operation and resnaps to chain truth on the next successful call. Its legit-revert window is real and bounded (MinterV1 covers arbitrum-one 5856338 → 6253359). No action proposed.

5 — bare calls. Aborting on revert is the preferable behavior for a determinism hazard. But a wrong-but-valid response still corrupts silently: a zero address from getNextTranscoderInPool exits the walk mid-list and every transcoder after it gets no Pool. Only event-derived membership fixes that.

Incident (instance of case 1)

Block 144056642, round 3154, transcoder 0xBAc7744ADA4AB1957CBaAFEF698B3c068bEB4fe0, tx 0xd308120046d57aac4a66d1466d68b602cb754e23a7c1d2dcbcadca8f91469081. Studio halted on pool!.rewardTokens with a null Pool. Network indexer QmUW5c… passed the same block on the same code, which is what identified it as indexer-side non-determinism. scripts/check-reward-pool.cjs confirms 99 transcoders in the on-chain pool at that block with the transcoder present, so a correct newRound() would have built the Pool.

Patched

#249 (merged) added createOrLoadPool() and routed newRound, reward, and ticketBroker through it, so a missing Pool is built on demand instead of dereferenced with !. Shipped with #245 (nullable Transaction.to).

Fixes the abort. Does not fix: the truncated enumeration (lazily-created Pools carry a different snapshot, and transcoders that neither rewarded nor redeemed fees still get none), POI divergence, or cases 2 and 3.

#246 and #247 are superseded by #245 and #249 and can be closed.

Next

Structural — derive pool membership from events (activation/deactivation/reward) instead of state reads, removing the enumeration entirely.

Hardening

• getTotalBonded: carry forward the last known protocol.totalActiveStake instead of zero(), mirroring the sane-fallback pattern already used for getGlobalTotalSupply.
• Raise log.info → log.error at roundsManager.ts:56,68. Both reverts are impossible post-launch, so info makes a determinism hazard invisible. log.critical would abort, which for POI divergence is arguably more honest — worth an explicit decision.
• Disambiguate the getPriceForPair zero sentinel.
• Use the existing safeDiv (helpers.ts:448) at livepeerToken.ts:43,95, where the guard checks the numerator but leaves totalSupply unguarded and BigDecimal.div by zero panics. Latent, not active — reaching it needs totalSupply == 0, unrealistic post-launch.

Operational — index against a consistent full-archive RPC, not a load-balanced pool that can hit a pruned or lagging backend.

Diagnosis

• scripts/check-reward-pool.cjs <rewardTx> — confirms whether a transcoder was in the on-chain pool at a block, isolating indexer-side anomalies. eth_call only, works against any archive RPC.
• docs/debugging.md — reproducing an indexing crash locally against Arbitrum from a chosen startBlock.

[linear-code]

ENG-192

[rickstaa]

A quick patch was added in #249. This can be reverted when we fix the root cause of the non-deterministic behavoir.