Skip to content

v1.3.9 — WaitGroup race fix + sync-before-consensus

Pre-release
Pre-release

Choose a tag to compare

@twobitapps twobitapps released this 10 Apr 01:17
· 28 commits to main since this release

v1.3.9 — WaitGroup race fix + sync-before-consensus

Two correctness bugs fixed. Both were necessary to break the
post-restart fork cascade documented in
POSTMORTEM_2026-04-09_FOUR_WAY_CHAIN_FORK.md.

Bug 1: BlockSTM sync.WaitGroup race

Symptom: validator crashed with

panic: sync: WaitGroup is reused before previous Wait has returned
goroutine 209226 [running]:
sync.(*WaitGroup).Wait(0xc05cc98670)
github.com/hyperspace-a1/hyperspace-edge/evm/parallel/blockstm.(*BlockSTMExecutor).ExecuteBlock.func1()
        /lambda/.../evm/parallel/blockstm/executor.go:145

Root cause: BlockSTMExecutor.workerWg lived as a field on the
executor struct and was reused across ExecuteBlock invocations. When
two ExecuteBlock calls overlapped (e.g. parallel proof verification,
or two near-simultaneous commits), the second call's workerWg.Add(N)
raced against the first call's workerWg.Wait(). The Go runtime
detects this and panics.

This panic was crashing val-1 ~once per day, and each crash → restart
triggered the post-restart sync race (Bug 2 below) → fork cascade.

Fix: workerWg is now a local variable declared inside
ExecuteBlock and passed to each worker via the worker() signature.
Concurrent ExecuteBlock calls each have their own WaitGroup. Tested
with go test -race.

Bug 2: sync runs in parallel with consensus on startup

Symptom: a freshly-restarted validator joins the network on its
own private fork. Sync logs show:

Block bodies processed: received=47 inserted=0
Failed to insert synced block: error="invalid block: invalid parent block"

because the local chain's parents don't match the network's parents.

Root cause: when a validator starts up, sync and consensus run in
parallel (p2p.Start() then ntConsensus.Start() immediately).
Cert quorum is satisfied locally as soon as 2 of 4 peers are connected
— there's no requirement that the local block chain matches the
network's. So the validator's bullshark immediately catches up via
view-change votes from peers, advances committedRound from 0 to
1500+, and produces blocks with empty parent sets that the rest of
the network can't link to. The chain forks.

The strict round-quorum fix in v1.3.8 didn't catch this because the
consensus engine was getting genuine 2f+1 view-change votes — those
are valid signatures. The issue is that bullshark catch-up shouldn't
even be running before the chain has caught up via sync.

Fix: New Sync.WaitForSync(maxWait, tolerance) method blocks
until localHead + tolerance >= maxPeerHead, with up to a 5-minute
deadline. cmd/server.go calls it between p2p.Start() and
ntConsensus.Start() so sync completes before consensus boots.
Liveness backstop: if no peers complete the StatusMsg handshake within
30 seconds, we proceed without waiting (otherwise the chain could
never bootstrap from genesis).

Verification

Both fixes verified end-to-end on the 4-validator Hyperspace A1 testnet
immediately after publication. The race-detector test suite is clean.

Upgrade notes

Drop-in upgrade. No chain data wipe required. Auto-updater pulls
this within 5 minutes. The WaitForSync gate adds up to ~5 minutes
to startup time on a node that needs to catch up many blocks; on a
node that's already caught up, the gate completes in seconds.

Files changed

  • evm/parallel/blockstm/executor.go — workerWg moved from struct
    field to local variable; worker signature updated
  • network/sync.go — new MaxPeerHead, PeerStatusCount, WaitForSync
    methods
  • cmd/server.go — call WaitForSync before ntConsensus.Start
  • main.go, cmd/version.go — version bump 1.3.8 → 1.3.9

Co-Authored-By: Claude Opus 4.6 (1M context)