Skip to content

v1.3.1 — Consensus Determinism Fix

Choose a tag to compare

@twobitapps twobitapps released this 09 Apr 16:20
· 28 commits to main since this release

Critical Consensus Determinism Fix

Fixes a four-way chain fork discovered in production on 2026-04-09 where each of
the 4 infrastructure validators was producing its own independent chain since
genesis, despite all starting from the same genesis block.

Verified in production: deployed to the 4-validator + 3-bootnode testnet;
all 4 validators now produce identical block hashes at every height.

Root causes

Three concurrent non-determinism bugs in NarwhalTusk consensus caused each
validator to derive a different block header from the same commit decision:

  1. Randomness beacon (randomness_beacon.go) — When BLS was disabled, the
    beacon fell back to hashing ECDSA vote signatures from the committed anchor.
    Different validators commit different anchors on different forks, so each
    derived a different leader-election seed and picked a different round
    leader. Fix: beacon now only publishes seeds when the anchor carries a
    BLS aggregate (which is deterministic regardless of which validator
    aggregates). In ECDSA-only mode, the beacon is empty and
    computeLeaderWithBeacon falls back to deterministicSeed(round), which
    depends only on the round number and is identical everywhere.

  2. Block timestamp (consensus.go) — Block.Time was set from
    time.Now(), stamping each validator's wall-clock onto the block header,
    so every validator produced a distinct hash from the same commit decision.
    Fix: blockTime = parent.Time + 1 — monotonic and deterministic.

  3. Block coinbase (consensus.go) — Every validator set the coinbase to
    its OWN address instead of the round leader, so block headers were unique
    by construction. Fix: coinbase is now the deterministic leader of the
    committed round, computed from the same round number and shared validator
    set on every node.

Additional fixes

  1. Optimistic wave skipping (bullshark.go) — When quorum couldn't be
    reached on a leader, each validator unilaterally skipped the wave based on
    LOCAL wall-clock elapsed time, producing divergent committed-round
    sequences. Disabled: waves only advance via the standard view-change
    path (2f+1 votes). If a validator can't reach quorum, consensus stalls —
    which is the correct BFT behaviour; producing different blocks on each node
    is actively worse than stalling.

  2. Bootstrap delay (consensus.go) — 30s settle delay before starting
    consensus so all validators can discover each other via bootnodes/DHT
    before producing certs. The previous 5s was insufficient — the slowest
    validator started consensus before the DAG was fully gossipped, producing
    its own empty blocks from genesis and creating a permanent fork.

Regression tests

  • TestPublishCommittedSeed_SkipsNonBLS — beacon refuses to publish seeds
    from ECDSA-only anchors.
  • TestLeaderElection_DeterministicWithoutBLS — 4 independent beacons
    committing divergent anchors still elect the same leader for every round.

Upgrade notes

Chain data MUST be wiped on upgrade — the pre-v1.3.1 chain was forked and has
no single canonical history. Operators running the auto-updater will receive
v1.3.1 automatically within 5 minutes; chain data wipe needs to be coordinated
with the infrastructure team.

Files changed

  • consensus/narwhaltusk/randomness_beacon.go
  • consensus/narwhaltusk/consensus.go
  • consensus/narwhaltusk/bullshark.go
  • consensus/narwhaltusk/randomness_beacon_test.go (new)
  • main.go, cmd/version.go (version bump 1.3.0 → 1.3.1)