Beacon node crash

[Maharacha]

🐞 Bug Report

Description

Beacon node crashed and restarted.
Version 3.2.2
Execution client is Erigon version 2.40.1-stable

Has this worked before in a previous version?

This was the first time it happened.

🔬 Minimal Reproduction

No relevant flags other than --execution-endpoint of course.

🔥 Error

prysm_error.log

🌍 Your Environment

Operating System:

LXC running Ubuntu 22.04.1 LTS

What version of Prysm are you running? (Which release)

3.2.2

Anything else relevant (validator index / public key)?

[prestonvanloon]

Thanks for this report. The most relevant part of your crash log is pasted below

fatal error: concurrent map read and map write
goroutine 310203778 [running]:
github.com/prysmaticlabs/prysm/v3/beacon-chain/forkchoice/doubly-linked-tree.(*ForkChoice).AncestorRoot(0xc0002ae580, {0xc076f8?, 0xc047cd67e0?}, {0x8f, 0xaf, 0x94, 0xcc, 0x19, 0x6, 0x5, ...}, ...)
        beacon-chain/forkchoice/doubly-linked-tree/forkchoice.go:263 +0x125
github.com/prysmaticlabs/prysm/v3/beacon-chain/blockchain.(*Service).ancestor(0xc00010f380, {0xc076f8?, 0xc047cd67b0?}, {0xc0251e71e0, 0x20, 0x6dcd7a27d0eb6ed5?}, 0xacdec631f4751eb5?)
        beacon-chain/blockchain/process_block_helpers.go:169 +0x1c3
github.com/prysmaticlabs/prysm/v3/beacon-chain/blockchain.(*Service).VerifyLmdFfgConsistency(0xc08fbfb180?, {0xc076f8, 0xc047cd67b0}, 0xc05ea08c60)
        beacon-chain/blockchain/receive_attestation.go:53 +0x7a
github.com/prysmaticlabs/prysm/v3/beacon-chain/sync.(*Service).validateAggregatedAtt(0xc01523ef00, {0xc076f8?, 0xc047cd66c0?}, 0xc0aac0ed70)
        beacon-chain/sync/validate_aggregate_proof.go:137 +0x12c
github.com/prysmaticlabs/prysm/v3/beacon-chain/sync.(*Service).validateAggregateAndProof(0xc01523ef00, {0xc076c0, 0xc05ea08ba0}, {0xc0a8231fb0, 0x27}, 0xc071692a80)
        beacon-chain/sync/validate_aggregate_proof.go:115 +0x785
github.com/prysmaticlabs/prysm/v3/beacon-chain/sync.(*Service).wrapAndReportValidation.func1({0xc07650?, 0xc003fe6d00?}, {0xc0a8231fb0, 0x27}, 0xc071692a80)
        beacon-chain/sync/subscriber.go:270 +0x3f3
github.com/libp2p/go-libp2p-pubsub.(*topicVal).validateMsg(0xc0457bc7b0, {0xc07650, 0xc003fe6d00}, {0xc0a8231fb0, 0x27}, 0xc00fe4f7a0?)
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:466 +0x137
github.com/libp2p/go-libp2p-pubsub.(*validation).validateSingleTopic(0xc01514a680, 0xc0457bc7b0, {0xc0a8231fb0, 0x27}, 0xc00fe4f7a0?)
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:444 +0x8d
github.com/libp2p/go-libp2p-pubsub.(*validation).validateTopic(0xc01514a6c0?, {0xc04a813198?, 0x7ab0cb?, 0x12?}, {0xc0a8231fb0?, 0x0?}, 0x33?)
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:393 +0xe5
github.com/libp2p/go-libp2p-pubsub.(*validation).doValidateTopic(0xc01514a680, {0xc04a813198?, 0xc182a0?, 0x1972725?}, {0xc0a8231fb0, 0x27}, 0xc071692a80, 0x0)
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:364 +0x5b
github.com/libp2p/go-libp2p-pubsub.(*validation).validate.func1()
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:329 +0x3e
created by github.com/libp2p/go-libp2p-pubsub.(*validation).validate
        external/com_github_libp2p_go_libp2p_pubsub/validation.go:328 +0x7f3
goroutine 1 [chan receive, 9988 minutes]:
github.com/prysmaticlabs/prysm/v3/beacon-chain/node.(*BeaconNode).Start(0xc0002adb80)
        beacon-chain/node/node.go:335 +0x1a8
main.startNode(0x193c507?)
        cmd/beacon-chain/main.go:297 +0x5ce

Looks related to recent forkchoice changes. cc @potuz

[prestonvanloon]

Concurrent map read/writes and this one seems to read without securing a read lock.

prysm/beacon-chain/forkchoice/doubly-linked-tree/forkchoice.go

Lines 259 to 263 in e2fa7d4

	func (f *ForkChoice) AncestorRoot(ctx context.Context, root [32]byte, slot primitives.Slot) ([32]byte, error) {
	ctx, span := trace.StartSpan(ctx, "doublyLinkedForkchoice.AncestorRoot")
	defer span.End()
	node, ok := f.store.nodeByRoot[root]

[Maharacha]

Wow that was taken care of super fast! Thanks!