v1.3.9 — WaitGroup race fix + sync-before-consensus
Pre-releasev1.3.9 — WaitGroup race fix + sync-before-consensus
Two correctness bugs fixed. Both were necessary to break the
post-restart fork cascade documented in
POSTMORTEM_2026-04-09_FOUR_WAY_CHAIN_FORK.md.
Bug 1: BlockSTM sync.WaitGroup race
Symptom: validator crashed with
panic: sync: WaitGroup is reused before previous Wait has returned
goroutine 209226 [running]:
sync.(*WaitGroup).Wait(0xc05cc98670)
github.com/hyperspace-a1/hyperspace-edge/evm/parallel/blockstm.(*BlockSTMExecutor).ExecuteBlock.func1()
/lambda/.../evm/parallel/blockstm/executor.go:145
Root cause: BlockSTMExecutor.workerWg lived as a field on the
executor struct and was reused across ExecuteBlock invocations. When
two ExecuteBlock calls overlapped (e.g. parallel proof verification,
or two near-simultaneous commits), the second call's workerWg.Add(N)
raced against the first call's workerWg.Wait(). The Go runtime
detects this and panics.
This panic was crashing val-1 ~once per day, and each crash → restart
triggered the post-restart sync race (Bug 2 below) → fork cascade.
Fix: workerWg is now a local variable declared inside
ExecuteBlock and passed to each worker via the worker() signature.
Concurrent ExecuteBlock calls each have their own WaitGroup. Tested
with go test -race.
Bug 2: sync runs in parallel with consensus on startup
Symptom: a freshly-restarted validator joins the network on its
own private fork. Sync logs show:
Block bodies processed: received=47 inserted=0
Failed to insert synced block: error="invalid block: invalid parent block"
because the local chain's parents don't match the network's parents.
Root cause: when a validator starts up, sync and consensus run in
parallel (p2p.Start() then ntConsensus.Start() immediately).
Cert quorum is satisfied locally as soon as 2 of 4 peers are connected
— there's no requirement that the local block chain matches the
network's. So the validator's bullshark immediately catches up via
view-change votes from peers, advances committedRound from 0 to
1500+, and produces blocks with empty parent sets that the rest of
the network can't link to. The chain forks.
The strict round-quorum fix in v1.3.8 didn't catch this because the
consensus engine was getting genuine 2f+1 view-change votes — those
are valid signatures. The issue is that bullshark catch-up shouldn't
even be running before the chain has caught up via sync.
Fix: New Sync.WaitForSync(maxWait, tolerance) method blocks
until localHead + tolerance >= maxPeerHead, with up to a 5-minute
deadline. cmd/server.go calls it between p2p.Start() and
ntConsensus.Start() so sync completes before consensus boots.
Liveness backstop: if no peers complete the StatusMsg handshake within
30 seconds, we proceed without waiting (otherwise the chain could
never bootstrap from genesis).
Verification
Both fixes verified end-to-end on the 4-validator Hyperspace A1 testnet
immediately after publication. The race-detector test suite is clean.
Upgrade notes
Drop-in upgrade. No chain data wipe required. Auto-updater pulls
this within 5 minutes. The WaitForSync gate adds up to ~5 minutes
to startup time on a node that needs to catch up many blocks; on a
node that's already caught up, the gate completes in seconds.
Files changed
evm/parallel/blockstm/executor.go— workerWg moved from struct
field to local variable; worker signature updatednetwork/sync.go— new MaxPeerHead, PeerStatusCount, WaitForSync
methodscmd/server.go— call WaitForSync before ntConsensus.Startmain.go,cmd/version.go— version bump 1.3.8 → 1.3.9
Co-Authored-By: Claude Opus 4.6 (1M context)