Symptom (reported by Beck, v1.8.4-m, mainnet, 2026-06-30)
PDPv0_Prove (task 4598096, dataset 1294) fails 5x in a row:
error: failed to submit possesion proof for dataset 1294: failed to get chain
randomness from beacon for pdp prove: chain: cannot draw randomness from future
epoch 6149477 (head 6149468)
Concurrent daemon stats:
[gossip-block] sub-rcv=125 ing-rcv=125 installed=11 skipped=43 backfilled=54 lastEpoch=6149489
[sync] polls=21 advances=1 reorgs=0 head=6149469 lastErr=""
Plus heavy bitswap protocols not supported / go away code 4101 peer-negotiation churn.
Root cause (two stacked bugs)
1. Bridge-off has no fallback for the future-epoch randomness draw.
ChainAPI.StateGetRandomnessDigestFromBeacon (rpc/handlers/chain_api.go ~L696) has a documented fallback for exactly the "local head is a few epochs behind the requested randEpoch" case — but the fallback is c.Bridge only. On the bridge-off path (the whole v1.8.x stock-Curio target) c.Bridge == nil, so it returns the local "future epoch" error immediately and the prove task burns its MaxFailures budget. The fallback that was designed for this case does not exist bridge-off.
2. Head lags the real tip under poor bitswap availability.
Gossip sees the tip at 6149489 but head is stuck at 6149469 (~20 behind), installed=11 vs skipped=43. Blocks arriving at head+N (N>1) are skipped and deferred to the polling Sync (net/blockingest), but Sync advances=1 over 22 polls — backfill is starving because bitswap can't find good peers (the protocols not supported / 4101 churn). So the head can't catch the tip, which is what puts the prove's randEpoch ahead of head in the first place.
Proposed fix
Bug 1 (priority — this is what fails the proof): Give the bridge-off path a real fallback for a randEpoch that is head < randEpoch <= head + smallWindow:
- Option A: short bounded wait-for-head — block up to ~N epochs / T seconds for the header sync to reach randEpoch, then draw locally. PDP randEpoch is at the proving-window open; the head is normally only 1-9 epochs behind, well within a short wait.
- Option B: trigger an on-demand head advance / targeted backfill to randEpoch before drawing.
- Either way: do not hard-fail bridge-off the instant the head is < randEpoch.
Bug 2 (head-lag robustness): Tighten head catch-up under poor bitswap: faster Sync cadence when gossip lastEpoch is well ahead of head, and/or escalate gossip-skipped head+N blocks to immediate targeted backfill instead of waiting for the next slow poll. Relates to #50 / #53 (bridge-off availability).
Acceptance
Reported by Beck (v1.8.4-m). Bridge-off parity tracking: #76.
Symptom (reported by Beck, v1.8.4-m, mainnet, 2026-06-30)
PDPv0_Prove(task 4598096, dataset 1294) fails 5x in a row:Concurrent daemon stats:
Plus heavy bitswap
protocols not supported/go away code 4101peer-negotiation churn.Root cause (two stacked bugs)
1. Bridge-off has no fallback for the future-epoch randomness draw.
ChainAPI.StateGetRandomnessDigestFromBeacon(rpc/handlers/chain_api.go ~L696) has a documented fallback for exactly the "local head is a few epochs behind the requested randEpoch" case — but the fallback isc.Bridgeonly. On the bridge-off path (the whole v1.8.x stock-Curio target)c.Bridge == nil, so it returns the local "future epoch" error immediately and the prove task burns its MaxFailures budget. The fallback that was designed for this case does not exist bridge-off.2. Head lags the real tip under poor bitswap availability.
Gossip sees the tip at 6149489 but head is stuck at 6149469 (~20 behind),
installed=11vsskipped=43. Blocks arriving at head+N (N>1) are skipped and deferred to the polling Sync (net/blockingest), but Syncadvances=1over 22 polls — backfill is starving because bitswap can't find good peers (theprotocols not supported/4101churn). So the head can't catch the tip, which is what puts the prove's randEpoch ahead of head in the first place.Proposed fix
Bug 1 (priority — this is what fails the proof): Give the bridge-off path a real fallback for a randEpoch that is
head < randEpoch <= head + smallWindow:Bug 2 (head-lag robustness): Tighten head catch-up under poor bitswap: faster Sync cadence when gossip lastEpoch is well ahead of head, and/or escalate gossip-skipped head+N blocks to immediate targeted backfill instead of waiting for the next slow poll. Relates to #50 / #53 (bridge-off availability).
Acceptance
Reported by Beck (v1.8.4-m). Bridge-off parity tracking: #76.