fatal error: concurrent map iteration and map write

[jsanders5]

System information

Geth version: 22.4.4
OS & Version: Debian 11

Expected behavior

Validator node does not crash.

Actual behavior

Validator node crashed and is unrecoverable.

Steps to reproduce the behavior

Not sure. The network had mined over 1M blocks before this error occurred. QBFT Network. Only 4 validator nodes and zero members. Validator smart contract from the example was deployed per the genesis file. This node was acting as an HTTP RPC node and a validator node.

Backtrace

Full log attached.

Snippet below:

address=0x676fD6F2974985AEA4c0C765d69fC483d9D46AE6 current.round=0 current.sequence=1033535 state=Preprepared
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: TRACE[09-21|22:57:20.078] QBFT: confirmation Formula used ceil(2N/3) address=0x676fD6F2974985AEA4c0C765d69fC483d9D46AE6 current.round=0 current.sequence=1033535 state=Preprepared
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: INFO [09-21|22:57:20.078] QBFT: received quorum of PREPARE messages address=0x676fD6F2974985AEA4c0C765d69fC483d9D46AE6 current.round=0 current.sequence=1033535 state=Preprepared      msg.code=19 msg.source=0x188C575838d052D160d734b6D0b96aa0Fb5a52Cc msg.round=0 msg.sequence=1033535 prepares.count=3 quorum=3
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: DEBUG[09-21|22:57:20.078] QBFT: PREPARE message matches proposal   address=0x676fD6F2974985AEA4c0C765d69fC483d9D46AE6 current.round=0 current.sequence=1033535 state=Preprepared      msg.code=19 msg.source=0x188C575838d052D160d734b6D0b96aa0Fb5a52Cc msg.round=0 msg.sequence=1033535 prepares.count=3 quorum=3 proposal="0558fc…fc2c9c" prepare="0558fc…fc2c9c"
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: INFO [09-21|22:57:20.078] QBFT: changed state                      address=0x676fD6F2974985AEA4c0C765d69fC483d9D46AE6 current.round=0 current.sequence=1033535 old.state=Preprepared      new.state=Prepared
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: TRACE[09-21|22:57:20.078] QBFT: process backlog                    address=0x676fD6F2974985AEA4c0C765d69fC483d9D46AE6 from=0x85B4DFD648B40c70afb85456DBc9a2FD01a3FF21 state=Prepared
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: TRACE[09-21|22:57:20.078] QBFT: process backlog                    address=0x676fD6F2974985AEA4c0C765d69fC483d9D46AE6 from=0x188C575838d052D160d734b6D0b96aa0Fb5a52Cc state=Prepared
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: INFO [09-21|22:57:20.078] QBFT: broadcast COMMIT message           address=0x676fD6F2974985AEA4c0C765d69fC483d9D46AE6 current.round=0 current.sequence=1033535 state=Prepared         msg.code=20 msg.source=0x676fD6F2974985AEA4c0C765d69fC483d9D46AE6 msg.round=0 msg.sequence=1033535 payload=0xf8aef869830fc53f80a00558fc2f177b2832b9f44dd8960d728eac6fb9e1d239bcce7751fc7129fc2c9cb841cc4532c202421345d2c5ba9afb2f97dc87a965304f4f23b8811671428feac91448444b049f3ae3342ab8994a6745ab51e99aa82a8e2ce39425ff4b7f4a7372fb00b8415f4b13cfe98570faea4c7277f3648ef6783ad778175c7d72d808d9a27473187a7a8df4f2b5753fea8a61eaf083e3f32c026dde5e06f290ffd27d0f64fc99467600
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: DEBUG[09-21|22:57:20.078] Ethereum quorum message handling failed  id=e72a4a4cd701b9d3 conn=inbound              err=EOF
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: DEBUG[09-21|22:57:20.078] Message handling failed in `eth`         id=e72a4a4cd701b9d3 conn=inbound              err=EOF
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: DEBUG[09-21|22:57:20.078] Removing Ethereum peer                   peer=e72a4a4c snap=true
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: TRACE[09-21|22:57:20.078] Unregistering sync peer                  peer=e72a4a4c
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: fatal error: concurrent map iteration and map write
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: goroutine 168 [running]:
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: runtime.throw(0x1891387, 0x26)
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]:         runtime/panic.go:1117 +0x72 fp=0xc15934fb58 sp=0xc15934fb28 pc=0x455ab2
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: runtime.mapiternext(0xc15934fc48)
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]:         runtime/map.go:858 +0x54c fp=0xc15934fbd8 sp=0xc15934fb58 pc=0x42e3ac
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: github.com/ethereum/go-ethereum/eth.(*handler).FindPeers(0xc000028d80, 0xc11b2f85a0, 0xc15934fd4c)
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]:         github.com/ethereum/go-ethereum/eth/handler.go:695 +0xa5 fp=0xc15934fcb8 sp=0xc15934fbd8 pc=0x102d685
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: github.com/ethereum/go-ethereum/consensus/istanbul/backend.(*Backend).Gossip(0xc00014e8c0, 0x1b39b60, 0xc16466f540, 0x13, 0xc000e7e770, 0x6c, 0x6c, 0x1, 0x0)
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]:         github.com/ethereum/go-ethereum/consensus/istanbul/backend/backend.go:178 +0x287 fp=0xc15934fec0 sp=0xc15934fcb8 pc=0xf2fc47
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: github.com/ethereum/go-ethereum/consensus/istanbul/qbft/core.(*core).handleEvents(0xc002050000)
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]:         github.com/ethereum/go-ethereum/consensus/istanbul/qbft/core/handler.go:127 +0x3dd fp=0xc15934ffd8 sp=0xc15934fec0 pc=0xf1c8dd
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: runtime.goexit()
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]:         runtime/asm_amd64.s:1371 +0x1 fp=0xc15934ffe0 sp=0xc15934ffd8 pc=0x48fde1
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: created by github.com/ethereum/go-ethereum/consensus/istanbul/qbft/core.(*core).Start
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]:         github.com/ethereum/go-ethereum/consensus/istanbul/qbft/core/handler.go:37 +0xac
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: goroutine 1 [chan receive, 8837 minutes]:
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: github.com/ethereum/go-ethereum/node.(*Node).Wait(...)
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]:         github.com/ethereum/go-ethereum/node/node.go:479
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: main.geth(0xc00027edc0, 0x0, 0x0)
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]:         github.com/ethereum/go-ethereum/cmd/geth/main.go:384 +0x157
Sep 21 22:57:20 timp-aws-v1 lace_run.sh[210070]: gopkg.in/urfave/cli%2ev1.HandleAction(0x15c3c40, 0x196f540, 0xc00027edc0, 0xc0005b97a0, 0x0)

nodeCrash_errorlog_20220921_225720.txt