feat: batch io operations when verifying & importing block

[twoeths]

Motivation

• We want to import block asap
• Persisting block may take up to 1s - 2s, in the worse scenario it could be up to 3s due to the I/O lag (see Investigate high execution payload verification time #5429)

Description

• We already have a Promise.all() in verifyBlock(), take that chance to persist blocks (and blobs) to db
• If there are errors, check if blocks are not in forkchoice then remove them from db
• Also process import block to LightClientServer in the next event loop to improve block time to head

Closes #5415

[github-actions]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: 0642605	Previous: c42adb2	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	569.57 us/op	719.78 us/op	0.79
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	48.208 us/op	47.044 us/op	1.02
BLS verify - blst-native	1.2116 ms/op	1.2190 ms/op	0.99
BLS verifyMultipleSignatures 3 - blst-native	2.4624 ms/op	2.4908 ms/op	0.99
BLS verifyMultipleSignatures 8 - blst-native	5.2883 ms/op	5.3351 ms/op	0.99
BLS verifyMultipleSignatures 32 - blst-native	19.290 ms/op	19.378 ms/op	1.00
BLS aggregatePubkeys 32 - blst-native	25.919 us/op	25.945 us/op	1.00
BLS aggregatePubkeys 128 - blst-native	100.55 us/op	101.14 us/op	0.99
getAttestationsForBlock	52.549 ms/op	56.323 ms/op	0.93
isKnown best case - 1 super set check	258.00 ns/op	248.00 ns/op	1.04
isKnown normal case - 2 super set checks	252.00 ns/op	250.00 ns/op	1.01
isKnown worse case - 16 super set checks	254.00 ns/op	245.00 ns/op	1.04
CheckpointStateCache - add get delete	4.8830 us/op	5.1390 us/op	0.95
validate gossip signedAggregateAndProof - struct	2.6946 ms/op	2.7767 ms/op	0.97
validate gossip attestation - struct	1.2805 ms/op	1.3253 ms/op	0.97
pickEth1Vote - no votes	1.2175 ms/op	1.3109 ms/op	0.93
pickEth1Vote - max votes	10.208 ms/op	10.613 ms/op	0.96
pickEth1Vote - Eth1Data hashTreeRoot value x2048	8.5587 ms/op	9.1202 ms/op	0.94
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	14.791 ms/op	14.443 ms/op	1.02
pickEth1Vote - Eth1Data fastSerialize value x2048	619.73 us/op	656.48 us/op	0.94
pickEth1Vote - Eth1Data fastSerialize tree x2048	6.7606 ms/op	4.6979 ms/op	1.44
bytes32 toHexString	499.00 ns/op	530.00 ns/op	0.94
bytes32 Buffer.toString(hex)	338.00 ns/op	381.00 ns/op	0.89
bytes32 Buffer.toString(hex) from Uint8Array	519.00 ns/op	603.00 ns/op	0.86
bytes32 Buffer.toString(hex) + 0x	325.00 ns/op	379.00 ns/op	0.86
Object access 1 prop	0.15300 ns/op	0.18200 ns/op	0.84
Map access 1 prop	0.14800 ns/op	0.16700 ns/op	0.89
Object get x1000	6.2300 ns/op	7.1950 ns/op	0.87
Map get x1000	0.58400 ns/op	0.64200 ns/op	0.91
Object set x1000	51.176 ns/op	55.971 ns/op	0.91
Map set x1000	42.137 ns/op	45.231 ns/op	0.93
Return object 10000 times	0.23150 ns/op	0.24140 ns/op	0.96
Throw Error 10000 times	4.1283 us/op	4.2326 us/op	0.98
fastMsgIdFn sha256 / 200 bytes	3.3870 us/op	3.4810 us/op	0.97
fastMsgIdFn h32 xxhash / 200 bytes	275.00 ns/op	297.00 ns/op	0.93
fastMsgIdFn h64 xxhash / 200 bytes	393.00 ns/op	416.00 ns/op	0.94
fastMsgIdFn sha256 / 1000 bytes	11.518 us/op	11.599 us/op	0.99
fastMsgIdFn h32 xxhash / 1000 bytes	404.00 ns/op	438.00 ns/op	0.92
fastMsgIdFn h64 xxhash / 1000 bytes	464.00 ns/op	508.00 ns/op	0.91
fastMsgIdFn sha256 / 10000 bytes	102.04 us/op	103.67 us/op	0.98
fastMsgIdFn h32 xxhash / 10000 bytes	1.8870 us/op	1.9680 us/op	0.96
fastMsgIdFn h64 xxhash / 10000 bytes	1.3400 us/op	1.4380 us/op	0.93
enrSubnets - fastDeserialize 64 bits	1.2510 us/op	1.3610 us/op	0.92
enrSubnets - ssz BitVector 64 bits	480.00 ns/op	518.00 ns/op	0.93
enrSubnets - fastDeserialize 4 bits	169.00 ns/op	186.00 ns/op	0.91
enrSubnets - ssz BitVector 4 bits	480.00 ns/op	561.00 ns/op	0.86
prioritizePeers score -10:0 att 32-0.1 sync 2-0	108.82 us/op	108.81 us/op	1.00
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	128.02 us/op	152.88 us/op	0.84
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	161.19 us/op	186.32 us/op	0.87
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	290.50 us/op	330.38 us/op	0.88
prioritizePeers score 0:0 att 64-1 sync 4-1	337.21 us/op	407.32 us/op	0.83
array of 16000 items push then shift	1.5446 us/op	1.6597 us/op	0.93
LinkedList of 16000 items push then shift	8.3790 ns/op	8.8020 ns/op	0.95
array of 16000 items push then pop	77.643 ns/op	108.99 ns/op	0.71
LinkedList of 16000 items push then pop	8.3020 ns/op	8.7900 ns/op	0.94
array of 24000 items push then shift	2.2985 us/op	2.4070 us/op	0.95
LinkedList of 24000 items push then shift	8.6380 ns/op	9.0320 ns/op	0.96
array of 24000 items push then pop	70.576 ns/op	86.522 ns/op	0.82
LinkedList of 24000 items push then pop	8.1140 ns/op	8.7000 ns/op	0.93
intersect bitArray bitLen 8	12.576 ns/op	13.227 ns/op	0.95
intersect array and set length 8	71.041 ns/op	83.127 ns/op	0.85
intersect bitArray bitLen 128	41.748 ns/op	43.915 ns/op	0.95
intersect array and set length 128	983.37 ns/op	1.1125 us/op	0.88
Buffer.concat 32 items	2.5090 us/op	2.8930 us/op	0.87
Uint8Array.set 32 items	2.7640 us/op	2.1850 us/op	1.26
pass gossip attestations to forkchoice per slot	2.6639 ms/op	3.0400 ms/op	0.88
computeDeltas	2.8050 ms/op	3.0063 ms/op	0.93
computeProposerBoostScoreFromBalances	1.7333 ms/op	1.7760 ms/op	0.98
altair processAttestation - 250000 vs - 7PWei normalcase	2.0543 ms/op	2.8171 ms/op	0.73
altair processAttestation - 250000 vs - 7PWei worstcase	3.1786 ms/op	3.9306 ms/op	0.81
altair processAttestation - setStatus - 1/6 committees join	137.10 us/op	149.34 us/op	0.92
altair processAttestation - setStatus - 1/3 committees join	269.28 us/op	284.35 us/op	0.95
altair processAttestation - setStatus - 1/2 committees join	361.82 us/op	364.48 us/op	0.99
altair processAttestation - setStatus - 2/3 committees join	450.68 us/op	463.05 us/op	0.97
altair processAttestation - setStatus - 4/5 committees join	606.71 us/op	649.22 us/op	0.93
altair processAttestation - setStatus - 100% committees join	745.37 us/op	768.31 us/op	0.97
altair processBlock - 250000 vs - 7PWei normalcase	19.137 ms/op	17.197 ms/op	1.11
altair processBlock - 250000 vs - 7PWei normalcase hashState	25.088 ms/op	24.724 ms/op	1.01
altair processBlock - 250000 vs - 7PWei worstcase	51.222 ms/op	51.144 ms/op	1.00
altair processBlock - 250000 vs - 7PWei worstcase hashState	68.020 ms/op	68.661 ms/op	0.99
phase0 processBlock - 250000 vs - 7PWei normalcase	1.9352 ms/op	2.1006 ms/op	0.92
phase0 processBlock - 250000 vs - 7PWei worstcase	27.250 ms/op	29.000 ms/op	0.94
altair processEth1Data - 250000 vs - 7PWei normalcase	461.83 us/op	488.74 us/op	0.94
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15	6.6880 us/op	9.2560 us/op	0.72
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219	19.726 us/op	31.102 us/op	0.63
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42	8.0940 us/op	13.880 us/op	0.58
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18	6.3310 us/op	9.3100 us/op	0.68
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020	73.915 us/op	115.80 us/op	0.64
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777	609.65 us/op	654.28 us/op	0.93
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384	872.67 us/op	904.19 us/op	0.97
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384	897.82 us/op	892.60 us/op	1.01
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384	2.1034 ms/op	2.4228 ms/op	0.87
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384	1.4687 ms/op	1.5835 ms/op	0.93
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384	3.6790 ms/op	4.1535 ms/op	0.89
Tree 40 250000 create	305.37 ms/op	352.69 ms/op	0.87
Tree 40 250000 get(125000)	178.19 ns/op	193.36 ns/op	0.92
Tree 40 250000 set(125000)	856.98 ns/op	1.0984 us/op	0.78
Tree 40 250000 toArray()	16.140 ms/op	22.054 ms/op	0.73
Tree 40 250000 iterate all - toArray() + loop	16.239 ms/op	22.247 ms/op	0.73
Tree 40 250000 iterate all - get(i)	62.570 ms/op	76.784 ms/op	0.81
MutableVector 250000 create	9.8399 ms/op	10.387 ms/op	0.95
MutableVector 250000 get(125000)	6.0390 ns/op	6.2540 ns/op	0.97
MutableVector 250000 set(125000)	248.31 ns/op	258.76 ns/op	0.96
MutableVector 250000 toArray()	3.0684 ms/op	2.9910 ms/op	1.03
MutableVector 250000 iterate all - toArray() + loop	2.9019 ms/op	3.0935 ms/op	0.94
MutableVector 250000 iterate all - get(i)	1.4746 ms/op	1.5076 ms/op	0.98
Array 250000 create	2.4819 ms/op	2.7521 ms/op	0.90
Array 250000 clone - spread	1.0890 ms/op	1.1386 ms/op	0.96
Array 250000 get(125000)	0.52800 ns/op	0.58300 ns/op	0.91
Array 250000 set(125000)	0.60200 ns/op	0.66700 ns/op	0.90
Array 250000 iterate all - loop	103.41 us/op	86.963 us/op	1.19
effectiveBalanceIncrements clone Uint8Array 300000	23.151 us/op	32.881 us/op	0.70
effectiveBalanceIncrements clone MutableVector 300000	322.00 ns/op	356.00 ns/op	0.90
effectiveBalanceIncrements rw all Uint8Array 300000	163.59 us/op	169.29 us/op	0.97
effectiveBalanceIncrements rw all MutableVector 300000	75.530 ms/op	83.924 ms/op	0.90
phase0 afterProcessEpoch - 250000 vs - 7PWei	108.31 ms/op	115.56 ms/op	0.94
phase0 beforeProcessEpoch - 250000 vs - 7PWei	40.639 ms/op	43.827 ms/op	0.93
altair processEpoch - mainnet_e81889	327.49 ms/op	330.33 ms/op	0.99
mainnet_e81889 - altair beforeProcessEpoch	64.369 ms/op	68.200 ms/op	0.94
mainnet_e81889 - altair processJustificationAndFinalization	16.005 us/op	20.890 us/op	0.77
mainnet_e81889 - altair processInactivityUpdates	5.5321 ms/op	6.0642 ms/op	0.91
mainnet_e81889 - altair processRewardsAndPenalties	52.588 ms/op	64.098 ms/op	0.82
mainnet_e81889 - altair processRegistryUpdates	2.6780 us/op	2.5900 us/op	1.03
mainnet_e81889 - altair processSlashings	449.00 ns/op	516.00 ns/op	0.87
mainnet_e81889 - altair processEth1DataReset	485.00 ns/op	645.00 ns/op	0.75
mainnet_e81889 - altair processEffectiveBalanceUpdates	1.2085 ms/op	1.2592 ms/op	0.96
mainnet_e81889 - altair processSlashingsReset	4.9440 us/op	4.8840 us/op	1.01
mainnet_e81889 - altair processRandaoMixesReset	5.5650 us/op	4.7670 us/op	1.17
mainnet_e81889 - altair processHistoricalRootsUpdate	916.00 ns/op	943.00 ns/op	0.97
mainnet_e81889 - altair processParticipationFlagUpdates	2.4800 us/op	4.2870 us/op	0.58
mainnet_e81889 - altair processSyncCommitteeUpdates	571.00 ns/op	694.00 ns/op	0.82
mainnet_e81889 - altair afterProcessEpoch	125.46 ms/op	128.35 ms/op	0.98
phase0 processEpoch - mainnet_e58758	359.14 ms/op	363.19 ms/op	0.99
mainnet_e58758 - phase0 beforeProcessEpoch	136.36 ms/op	125.21 ms/op	1.09
mainnet_e58758 - phase0 processJustificationAndFinalization	17.659 us/op	21.915 us/op	0.81
mainnet_e58758 - phase0 processRewardsAndPenalties	64.858 ms/op	56.374 ms/op	1.15
mainnet_e58758 - phase0 processRegistryUpdates	7.8190 us/op	8.2340 us/op	0.95
mainnet_e58758 - phase0 processSlashings	478.00 ns/op	1.5760 us/op	0.30
mainnet_e58758 - phase0 processEth1DataReset	476.00 ns/op	715.00 ns/op	0.67
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	957.43 us/op	1.0337 ms/op	0.93
mainnet_e58758 - phase0 processSlashingsReset	3.4340 us/op	4.1330 us/op	0.83
mainnet_e58758 - phase0 processRandaoMixesReset	4.8720 us/op	5.1500 us/op	0.95
mainnet_e58758 - phase0 processHistoricalRootsUpdate	602.00 ns/op	734.00 ns/op	0.82
mainnet_e58758 - phase0 processParticipationRecordUpdates	3.7390 us/op	4.4860 us/op	0.83
mainnet_e58758 - phase0 afterProcessEpoch	96.941 ms/op	99.006 ms/op	0.98
phase0 processEffectiveBalanceUpdates - 250000 normalcase	1.2250 ms/op	1.2539 ms/op	0.98
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	1.5900 ms/op	1.6424 ms/op	0.97
altair processInactivityUpdates - 250000 normalcase	25.097 ms/op	21.101 ms/op	1.19
altair processInactivityUpdates - 250000 worstcase	26.245 ms/op	25.502 ms/op	1.03
phase0 processRegistryUpdates - 250000 normalcase	6.6070 us/op	15.596 us/op	0.42
phase0 processRegistryUpdates - 250000 badcase_full_deposits	235.46 us/op	304.62 us/op	0.77
phase0 processRegistryUpdates - 250000 worstcase 0.5	129.35 ms/op	118.31 ms/op	1.09
altair processRewardsAndPenalties - 250000 normalcase	66.520 ms/op	62.618 ms/op	1.06
altair processRewardsAndPenalties - 250000 worstcase	73.508 ms/op	65.284 ms/op	1.13
phase0 getAttestationDeltas - 250000 normalcase	6.6922 ms/op	7.1462 ms/op	0.94
phase0 getAttestationDeltas - 250000 worstcase	6.5351 ms/op	6.8557 ms/op	0.95
phase0 processSlashings - 250000 worstcase	3.5934 ms/op	3.4050 ms/op	1.06
altair processSyncCommitteeUpdates - 250000	181.39 ms/op	180.60 ms/op	1.00
BeaconState.hashTreeRoot - No change	323.00 ns/op	273.00 ns/op	1.18
BeaconState.hashTreeRoot - 1 full validator	53.634 us/op	54.256 us/op	0.99
BeaconState.hashTreeRoot - 32 full validator	559.83 us/op	514.56 us/op	1.09
BeaconState.hashTreeRoot - 512 full validator	5.2729 ms/op	5.4991 ms/op	0.96
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	60.946 us/op	65.080 us/op	0.94
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	887.84 us/op	868.85 us/op	1.02
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	11.130 ms/op	11.360 ms/op	0.98
BeaconState.hashTreeRoot - 1 balances	49.930 us/op	46.665 us/op	1.07
BeaconState.hashTreeRoot - 32 balances	465.31 us/op	465.34 us/op	1.00
BeaconState.hashTreeRoot - 512 balances	4.5609 ms/op	4.3749 ms/op	1.04
BeaconState.hashTreeRoot - 250000 balances	75.911 ms/op	74.166 ms/op	1.02
aggregationBits - 2048 els - zipIndexesInBitList	15.488 us/op	17.806 us/op	0.87
regular array get 100000 times	35.694 us/op	33.095 us/op	1.08
wrappedArray get 100000 times	35.390 us/op	33.043 us/op	1.07
arrayWithProxy get 100000 times	15.956 ms/op	15.694 ms/op	1.02
ssz.Root.equals	537.00 ns/op	606.00 ns/op	0.89
byteArrayEquals	534.00 ns/op	563.00 ns/op	0.95
shuffle list - 16384 els	6.9967 ms/op	6.9243 ms/op	1.01
shuffle list - 250000 els	101.13 ms/op	101.66 ms/op	0.99
processSlot - 1 slots	9.0980 us/op	9.1680 us/op	0.99
processSlot - 32 slots	1.4063 ms/op	1.3723 ms/op	1.02
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	36.518 ms/op	35.870 ms/op	1.02
getCommitteeAssignments - req 1 vs - 250000 vc	2.8799 ms/op	2.9227 ms/op	0.99
getCommitteeAssignments - req 100 vs - 250000 vc	4.1376 ms/op	4.1499 ms/op	1.00
getCommitteeAssignments - req 1000 vs - 250000 vc	4.4733 ms/op	4.5131 ms/op	0.99
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	4.9700 ns/op	4.7800 ns/op	1.04
state getBlockRootAtSlot - 250000 vs - 7PWei	906.27 ns/op	621.29 ns/op	1.46
computeProposers - vc 250000	10.846 ms/op	10.362 ms/op	1.05
computeEpochShuffling - vc 250000	101.70 ms/op	103.23 ms/op	0.99
getNextSyncCommittee - vc 250000	180.46 ms/op	174.33 ms/op	1.04
computeSigningRoot for AttestationData	13.936 us/op	13.378 us/op	1.04
hash AttestationData serialized data then Buffer.toString(base64)	2.4831 us/op	2.4777 us/op	1.00
toHexString serialized data	1.0714 us/op	1.1070 us/op	0.97
Buffer.toString(base64)	319.67 ns/op	347.43 ns/op	0.92

by benchmarkbot/action

[dapplion]

Pushed some changes, see

lodestar/packages/beacon-node/src/chain/blocks/index.ts

Lines 145 to 159 in 4ce2967

	// Clean db if we don't have blocks in forkchoice but already persisted them to db
	//
	// NOTE: this function is awaited to ensure that DB size remains constant, otherwise an attacker may bloat the
	// disk with big malicious payloads. Our sequential block importer will wait for this promise before importing
	// another block. The removal call error is not propagated since that would halt the chain.
	//
	// LOG: Because the error is not propagated and there's a risk of db bloat, the error is logged at warn level
	// to alert the user of potential db bloat. This error _should_ never happen user must act and report to us
	await removeEagerlyPeristedBlockInputs.call(this, blocks).catch((e) => {
	this.logger.warn(
	"Error pruning eagerly imported block inputs, DB may grow in size if this error happens frequently",
	{slot: blocks.map((block) => block.block.message.slot).join(",")},
	e
	);
	});

@tuyennhv we have to be careful here since this may open up attacks. We are exposing DB growth to potentially untrusted data. Blocks from gossip have proposer sig validated but blocks from sync have no validation at all. We may want to disable this feature for sync and only do for payloads with some validation like gossip or unknown block sync

[twoeths]

@dapplion I added eagerPersistBlock flag to ImportBlockOpts so that only blocks coming from gossip or unknown block sync gets this feature

[g11tech]

We

agreed, anyway eager saving on sync makes no effective impact because of checkpoint sync

[dapplion]

Looks great! Both gossip and unknown block root are safe for eagerPersistBlock. Can you open an issue to add a test that asserts that unknown block sync won't send to the chain blocks with invalid root?

[twoeths]

@dapplion I added that test case to this commit c425a24

see #5485

[wemeetagain]

🎉 This PR is included in v1.9.0 🎉