fix: prune checkpoint states in processSlotsToNearestCheckpoint

[twoeths]

Motivation

• in holesky, there was a single block killing lodestar because we have to process multiple epoch transitions at the same time, see Not able to prune checkpoint state during nft of holesky #7495 (comment)

Description

• for single epoch transition, let the next block prune it at the last 1/3 slot of next epoch
• if there are multiple epoch transitions, it means network is not healthy so prune it right away or we'll get OOM
• also fix "Can only get block root in the past" in Not able to prune checkpoint state during nft of holesky #7495

Closes #7495

Steps to test or reproduce

Test holesky

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 31 lines in your changes missing coverage. Please review.

Project coverage is 50.42%. Comparing base (d1f9f7b) to head (808b19e).
Report is 2 commits behind head on unstable.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #7497      +/-   ##
============================================
- Coverage     50.47%   50.42%   -0.05%     
============================================
  Files           603      603              
  Lines         40661    40686      +25     
  Branches       2229     2226       -3     
============================================
- Hits          20523    20517       -6     
- Misses        20098    20129      +31     
  Partials         40       40              

[github-actions]

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite	Current: 74fb846	Previous: b2ebe49	Ratio
altair processBlock - 250000 vs - 7PWei normalcase	13.623 ms/op	3.2878 ms/op	4.14
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	15.315 us/op	4.7920 us/op	3.20
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	2.3343 ms/op	749.66 us/op	3.11
Array.fill - length 1000000	9.8681 ms/op	2.5848 ms/op	3.82
Array push - length 1000000	86.720 ms/op	10.026 ms/op	8.65
phase0 getAttestationDeltas - 250000 worstcase	18.823 ms/op	5.0277 ms/op	3.74

Full benchmark results

Benchmark suite	Current: 74fb846	Previous: b2ebe49	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	1.0000 ms/op	821.60 us/op	1.22
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	39.492 us/op	36.909 us/op	1.07
BLS verify - blst	845.49 us/op	846.41 us/op	1.00
BLS verifyMultipleSignatures 3 - blst	1.2551 ms/op	1.2985 ms/op	0.97
BLS verifyMultipleSignatures 8 - blst	1.9649 ms/op	1.7043 ms/op	1.15
BLS verifyMultipleSignatures 32 - blst	6.4997 ms/op	4.7000 ms/op	1.38
BLS verifyMultipleSignatures 64 - blst	11.413 ms/op	8.5167 ms/op	1.34
BLS verifyMultipleSignatures 128 - blst	18.177 ms/op	16.894 ms/op	1.08
BLS deserializing 10000 signatures	713.18 ms/op	651.26 ms/op	1.10
BLS deserializing 100000 signatures	7.5778 s/op	6.5274 s/op	1.16
BLS verifyMultipleSignatures - same message - 3 - blst	1.1657 ms/op	973.25 us/op	1.20
BLS verifyMultipleSignatures - same message - 8 - blst	1.4063 ms/op	1.0973 ms/op	1.28
BLS verifyMultipleSignatures - same message - 32 - blst	2.1542 ms/op	1.6573 ms/op	1.30
BLS verifyMultipleSignatures - same message - 64 - blst	3.0589 ms/op	2.5560 ms/op	1.20
BLS verifyMultipleSignatures - same message - 128 - blst	5.1609 ms/op	4.2679 ms/op	1.21
BLS aggregatePubkeys 32 - blst	22.534 us/op	18.118 us/op	1.24
BLS aggregatePubkeys 128 - blst	79.467 us/op	63.979 us/op	1.24
notSeenSlots=1 numMissedVotes=1 numBadVotes=10	61.691 ms/op	45.748 ms/op	1.35
notSeenSlots=1 numMissedVotes=0 numBadVotes=4	63.390 ms/op	49.358 ms/op	1.28
notSeenSlots=2 numMissedVotes=1 numBadVotes=10	49.587 ms/op	42.040 ms/op	1.18
getSlashingsAndExits - default max	96.465 us/op	57.799 us/op	1.67
getSlashingsAndExits - 2k	376.62 us/op	394.12 us/op	0.96
proposeBlockBody type=full, size=empty	5.4095 ms/op	5.7180 ms/op	0.95
isKnown best case - 1 super set check	217.00 ns/op	416.00 ns/op	0.52
isKnown normal case - 2 super set checks	222.00 ns/op	417.00 ns/op	0.53
isKnown worse case - 16 super set checks	219.00 ns/op	415.00 ns/op	0.53
InMemoryCheckpointStateCache - add get delete	2.8320 us/op	2.6260 us/op	1.08
validate api signedAggregateAndProof - struct	1.6146 ms/op	1.7770 ms/op	0.91
validate gossip signedAggregateAndProof - struct	1.6077 ms/op	1.6909 ms/op	0.95
batch validate gossip attestation - vc 640000 - chunk 32	143.01 us/op	117.35 us/op	1.22
batch validate gossip attestation - vc 640000 - chunk 64	125.96 us/op	101.69 us/op	1.24
batch validate gossip attestation - vc 640000 - chunk 128	123.25 us/op	90.480 us/op	1.36
batch validate gossip attestation - vc 640000 - chunk 256	132.19 us/op	92.539 us/op	1.43
pickEth1Vote - no votes	1.1741 ms/op	726.69 us/op	1.62
pickEth1Vote - max votes	12.155 ms/op	4.5820 ms/op	2.65
pickEth1Vote - Eth1Data hashTreeRoot value x2048	22.756 ms/op	10.626 ms/op	2.14
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	27.826 ms/op	16.875 ms/op	1.65
pickEth1Vote - Eth1Data fastSerialize value x2048	484.19 us/op	332.32 us/op	1.46
pickEth1Vote - Eth1Data fastSerialize tree x2048	4.9269 ms/op	1.7991 ms/op	2.74
bytes32 toHexString	419.00 ns/op	497.00 ns/op	0.84
bytes32 Buffer.toString(hex)	285.00 ns/op	407.00 ns/op	0.70
bytes32 Buffer.toString(hex) from Uint8Array	369.00 ns/op	479.00 ns/op	0.77
bytes32 Buffer.toString(hex) + 0x	289.00 ns/op	415.00 ns/op	0.70
Object access 1 prop	0.13000 ns/op	0.29700 ns/op	0.44
Map access 1 prop	0.14400 ns/op	0.30400 ns/op	0.47
Object get x1000	6.1660 ns/op	4.8510 ns/op	1.27
Map get x1000	6.9680 ns/op	5.4550 ns/op	1.28
Object set x1000	31.538 ns/op	21.622 ns/op	1.46
Map set x1000	21.594 ns/op	17.727 ns/op	1.22
Return object 10000 times	0.30330 ns/op	0.29070 ns/op	1.04
Throw Error 10000 times	4.6995 us/op	3.6235 us/op	1.30
toHex	161.90 ns/op	96.193 ns/op	1.68
Buffer.from	140.26 ns/op	88.269 ns/op	1.59
shared Buffer	85.486 ns/op	61.137 ns/op	1.40
fastMsgIdFn sha256 / 200 bytes	2.4000 us/op	1.8220 us/op	1.32
fastMsgIdFn h32 xxhash / 200 bytes	224.00 ns/op	363.00 ns/op	0.62
fastMsgIdFn h64 xxhash / 200 bytes	341.00 ns/op	448.00 ns/op	0.76
fastMsgIdFn sha256 / 1000 bytes	8.4850 us/op	5.8940 us/op	1.44
fastMsgIdFn h32 xxhash / 1000 bytes	385.00 ns/op	523.00 ns/op	0.74
fastMsgIdFn h64 xxhash / 1000 bytes	409.00 ns/op	533.00 ns/op	0.77
fastMsgIdFn sha256 / 10000 bytes	71.164 us/op	50.573 us/op	1.41
fastMsgIdFn h32 xxhash / 10000 bytes	2.2190 us/op	1.9370 us/op	1.15
fastMsgIdFn h64 xxhash / 10000 bytes	1.3200 us/op	1.3650 us/op	0.97
send data - 1000 256B messages	17.616 ms/op	14.549 ms/op	1.21
send data - 1000 512B messages	18.510 ms/op	17.834 ms/op	1.04
send data - 1000 1024B messages	32.270 ms/op	20.993 ms/op	1.54
send data - 1000 1200B messages	28.942 ms/op	15.835 ms/op	1.83
send data - 1000 2048B messages	33.471 ms/op	17.529 ms/op	1.91
send data - 1000 4096B messages	39.968 ms/op	19.786 ms/op	2.02
send data - 1000 16384B messages	78.331 ms/op	48.079 ms/op	1.63
send data - 1000 65536B messages	249.20 ms/op	274.80 ms/op	0.91
enrSubnets - fastDeserialize 64 bits	977.00 ns/op	1.6920 us/op	0.58
enrSubnets - ssz BitVector 64 bits	347.00 ns/op	497.00 ns/op	0.70
enrSubnets - fastDeserialize 4 bits	150.00 ns/op	316.00 ns/op	0.47
enrSubnets - ssz BitVector 4 bits	345.00 ns/op	496.00 ns/op	0.70
prioritizePeers score -10:0 att 32-0.1 sync 2-0	134.16 us/op	102.45 us/op	1.31
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	157.82 us/op	122.09 us/op	1.29
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	247.98 us/op	176.56 us/op	1.40
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	417.91 us/op	316.13 us/op	1.32
prioritizePeers score 0:0 att 64-1 sync 4-1	512.11 us/op	410.41 us/op	1.25
array of 16000 items push then shift	1.7188 us/op	1.3212 us/op	1.30
LinkedList of 16000 items push then shift	8.0900 ns/op	6.7010 ns/op	1.21
array of 16000 items push then pop	86.056 ns/op	70.809 ns/op	1.22
LinkedList of 16000 items push then pop	7.8470 ns/op	6.3230 ns/op	1.24
array of 24000 items push then shift	2.5336 us/op	1.9451 us/op	1.30
LinkedList of 24000 items push then shift	7.7450 ns/op	6.4680 ns/op	1.20
array of 24000 items push then pop	118.16 ns/op	94.515 ns/op	1.25
LinkedList of 24000 items push then pop	10.279 ns/op	6.3340 ns/op	1.62
intersect bitArray bitLen 8	6.5380 ns/op	5.6940 ns/op	1.15
intersect array and set length 8	40.283 ns/op	33.164 ns/op	1.21
intersect bitArray bitLen 128	30.604 ns/op	27.396 ns/op	1.12
intersect array and set length 128	677.10 ns/op	559.11 ns/op	1.21
bitArray.getTrueBitIndexes() bitLen 128	1.0910 us/op	1.2080 us/op	0.90
bitArray.getTrueBitIndexes() bitLen 248	2.1740 us/op	1.9580 us/op	1.11
bitArray.getTrueBitIndexes() bitLen 512	4.2220 us/op	3.7540 us/op	1.12
Buffer.concat 32 items	705.00 ns/op	854.00 ns/op	0.83
Uint8Array.set 32 items	1.8460 us/op	1.0770 us/op	1.71
Buffer.copy	3.2280 us/op	2.0250 us/op	1.59
Uint8Array.set - with subarray	3.0820 us/op	1.6480 us/op	1.87
Uint8Array.set - without subarray	2.8850 us/op	1.0970 us/op	2.63
getUint32 - dataview	216.00 ns/op	382.00 ns/op	0.57
getUint32 - manual	137.00 ns/op	322.00 ns/op	0.43
Set add up to 64 items then delete first	3.0668 us/op	1.8556 us/op	1.65
OrderedSet add up to 64 items then delete first	4.6563 us/op	2.8647 us/op	1.63
Set add up to 64 items then delete last	3.4879 us/op	2.1261 us/op	1.64
OrderedSet add up to 64 items then delete last	4.5324 us/op	3.1523 us/op	1.44
Set add up to 64 items then delete middle	3.1917 us/op	2.0963 us/op	1.52
OrderedSet add up to 64 items then delete middle	6.7491 us/op	4.6612 us/op	1.45
Set add up to 128 items then delete first	6.6246 us/op	4.1403 us/op	1.60
OrderedSet add up to 128 items then delete first	10.618 us/op	6.5427 us/op	1.62
Set add up to 128 items then delete last	6.7838 us/op	4.0160 us/op	1.69
OrderedSet add up to 128 items then delete last	11.370 us/op	6.2300 us/op	1.83
Set add up to 128 items then delete middle	6.1989 us/op	4.0610 us/op	1.53
OrderedSet add up to 128 items then delete middle	17.879 us/op	12.321 us/op	1.45
Set add up to 256 items then delete first	13.669 us/op	8.2304 us/op	1.66
OrderedSet add up to 256 items then delete first	21.069 us/op	13.245 us/op	1.59
Set add up to 256 items then delete last	13.662 us/op	7.9431 us/op	1.72
OrderedSet add up to 256 items then delete last	23.349 us/op	12.153 us/op	1.92
Set add up to 256 items then delete middle	14.630 us/op	7.8783 us/op	1.86
OrderedSet add up to 256 items then delete middle	51.866 us/op	37.128 us/op	1.40
transfer serialized Status (84 B)	3.3240 us/op	2.1750 us/op	1.53
copy serialized Status (84 B)	2.2290 us/op	1.3510 us/op	1.65
transfer serialized SignedVoluntaryExit (112 B)	3.2750 us/op	2.1270 us/op	1.54
copy serialized SignedVoluntaryExit (112 B)	1.9780 us/op	1.3710 us/op	1.44
transfer serialized ProposerSlashing (416 B)	4.0760 us/op	2.4300 us/op	1.68
copy serialized ProposerSlashing (416 B)	2.9030 us/op	2.0530 us/op	1.41
transfer serialized Attestation (485 B)	4.2060 us/op	2.5280 us/op	1.66
copy serialized Attestation (485 B)	2.9950 us/op	1.7480 us/op	1.71
transfer serialized AttesterSlashing (33232 B)	3.8420 us/op	2.9250 us/op	1.31
copy serialized AttesterSlashing (33232 B)	5.9280 us/op	3.1930 us/op	1.86
transfer serialized Small SignedBeaconBlock (128000 B)	3.9180 us/op	2.7210 us/op	1.44
copy serialized Small SignedBeaconBlock (128000 B)	16.625 us/op	7.5990 us/op	2.19
transfer serialized Avg SignedBeaconBlock (200000 B)	4.9060 us/op	3.0080 us/op	1.63
copy serialized Avg SignedBeaconBlock (200000 B)	22.678 us/op	12.755 us/op	1.78
transfer serialized BlobsSidecar (524380 B)	5.0860 us/op	3.9080 us/op	1.30
copy serialized BlobsSidecar (524380 B)	71.764 us/op	58.505 us/op	1.23
transfer serialized Big SignedBeaconBlock (1000000 B)	5.3570 us/op	4.4420 us/op	1.21
copy serialized Big SignedBeaconBlock (1000000 B)	165.28 us/op	306.84 us/op	0.54
pass gossip attestations to forkchoice per slot	3.0654 ms/op	3.0781 ms/op	1.00
forkChoice updateHead vc 100000 bc 64 eq 0	517.39 us/op	352.16 us/op	1.47
forkChoice updateHead vc 600000 bc 64 eq 0	3.2737 ms/op	2.2102 ms/op	1.48
forkChoice updateHead vc 1000000 bc 64 eq 0	5.8095 ms/op	4.0414 ms/op	1.44
forkChoice updateHead vc 600000 bc 320 eq 0	3.2969 ms/op	2.4632 ms/op	1.34
forkChoice updateHead vc 600000 bc 1200 eq 0	3.3416 ms/op	2.4895 ms/op	1.34
forkChoice updateHead vc 600000 bc 7200 eq 0	3.5802 ms/op	2.4449 ms/op	1.46
forkChoice updateHead vc 600000 bc 64 eq 1000	11.321 ms/op	9.7549 ms/op	1.16
forkChoice updateHead vc 600000 bc 64 eq 10000	11.335 ms/op	9.8799 ms/op	1.15
forkChoice updateHead vc 600000 bc 64 eq 300000	14.741 ms/op	11.724 ms/op	1.26
computeDeltas 500000 validators 300 proto nodes	4.2663 ms/op	3.2737 ms/op	1.30
computeDeltas 500000 validators 1200 proto nodes	4.3539 ms/op	3.3626 ms/op	1.29
computeDeltas 500000 validators 7200 proto nodes	4.3044 ms/op	3.4099 ms/op	1.26
computeDeltas 750000 validators 300 proto nodes	6.9773 ms/op	5.0297 ms/op	1.39
computeDeltas 750000 validators 1200 proto nodes	7.2736 ms/op	5.0063 ms/op	1.45
computeDeltas 750000 validators 7200 proto nodes	6.8555 ms/op	4.9812 ms/op	1.38
computeDeltas 1400000 validators 300 proto nodes	12.320 ms/op	9.4721 ms/op	1.30
computeDeltas 1400000 validators 1200 proto nodes	13.098 ms/op	9.9900 ms/op	1.31
computeDeltas 1400000 validators 7200 proto nodes	12.575 ms/op	9.8459 ms/op	1.28
computeDeltas 2100000 validators 300 proto nodes	19.529 ms/op	14.221 ms/op	1.37
computeDeltas 2100000 validators 1200 proto nodes	20.618 ms/op	13.460 ms/op	1.53
computeDeltas 2100000 validators 7200 proto nodes	20.813 ms/op	13.822 ms/op	1.51
altair processAttestation - 250000 vs - 7PWei normalcase	4.3860 ms/op	1.7073 ms/op	2.57
altair processAttestation - 250000 vs - 7PWei worstcase	4.9751 ms/op	2.4727 ms/op	2.01
altair processAttestation - setStatus - 1/6 committees join	188.18 us/op	100.73 us/op	1.87
altair processAttestation - setStatus - 1/3 committees join	296.22 us/op	216.97 us/op	1.37
altair processAttestation - setStatus - 1/2 committees join	485.44 us/op	311.81 us/op	1.56
altair processAttestation - setStatus - 2/3 committees join	1.0087 ms/op	383.76 us/op	2.63
altair processAttestation - setStatus - 4/5 committees join	1.0430 ms/op	488.71 us/op	2.13
altair processAttestation - setStatus - 100% committees join	965.00 us/op	635.47 us/op	1.52
altair processBlock - 250000 vs - 7PWei normalcase	13.623 ms/op	3.2878 ms/op	4.14
altair processBlock - 250000 vs - 7PWei normalcase hashState	49.546 ms/op	32.707 ms/op	1.51
altair processBlock - 250000 vs - 7PWei worstcase	67.379 ms/op	41.330 ms/op	1.63
altair processBlock - 250000 vs - 7PWei worstcase hashState	158.62 ms/op	83.363 ms/op	1.90
phase0 processBlock - 250000 vs - 7PWei normalcase	3.7673 ms/op	1.5308 ms/op	2.46
phase0 processBlock - 250000 vs - 7PWei worstcase	36.595 ms/op	26.403 ms/op	1.39
altair processEth1Data - 250000 vs - 7PWei normalcase	522.76 us/op	278.98 us/op	1.87
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15	12.411 us/op	4.4700 us/op	2.78
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	49.743 us/op	28.797 us/op	1.73
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42	18.409 us/op	7.4480 us/op	2.47
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	15.315 us/op	4.7920 us/op	3.20
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020	175.01 us/op	111.51 us/op	1.57
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	2.3343 ms/op	749.66 us/op	3.11
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	3.0083 ms/op	1.0395 ms/op	2.89
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	2.9007 ms/op	1.0484 ms/op	2.77
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	5.5207 ms/op	2.9429 ms/op	1.88
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	2.5166 ms/op	1.0048 ms/op	2.50
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	8.7328 ms/op	2.9891 ms/op	2.92
Tree 40 250000 create	816.76 ms/op	372.14 ms/op	2.19
Tree 40 250000 get(125000)	209.74 ns/op	115.15 ns/op	1.82
Tree 40 250000 set(125000)	4.1709 us/op	1.2184 us/op	3.42
Tree 40 250000 toArray()	32.335 ms/op	9.8112 ms/op	3.30
Tree 40 250000 iterate all - toArray() + loop	31.532 ms/op	9.7119 ms/op	3.25
Tree 40 250000 iterate all - get(i)	82.367 ms/op	40.436 ms/op	2.04
Array 250000 create	5.3347 ms/op	2.4717 ms/op	2.16
Array 250000 clone - spread	3.2720 ms/op	648.92 us/op	5.04
Array 250000 get(125000)	0.49700 ns/op	0.56300 ns/op	0.88
Array 250000 set(125000)	0.52300 ns/op	0.57500 ns/op	0.91
Array 250000 iterate all - loop	128.05 us/op	77.221 us/op	1.66
phase0 afterProcessEpoch - 250000 vs - 7PWei	67.055 ms/op	41.197 ms/op	1.63
Array.fill - length 1000000	9.8681 ms/op	2.5848 ms/op	3.82
Array push - length 1000000	86.720 ms/op	10.026 ms/op	8.65
Array.get	0.33410 ns/op	0.26256 ns/op	1.27
Uint8Array.get	0.78999 ns/op	0.34081 ns/op	2.32
phase0 beforeProcessEpoch - 250000 vs - 7PWei	35.164 ms/op	13.119 ms/op	2.68
altair processEpoch - mainnet_e81889	484.36 ms/op	247.50 ms/op	1.96
mainnet_e81889 - altair beforeProcessEpoch	30.968 ms/op	15.869 ms/op	1.95
mainnet_e81889 - altair processJustificationAndFinalization	9.9410 us/op	5.1610 us/op	1.93
mainnet_e81889 - altair processInactivityUpdates	7.9046 ms/op	3.7017 ms/op	2.14
mainnet_e81889 - altair processRewardsAndPenalties	52.698 ms/op	36.252 ms/op	1.45
mainnet_e81889 - altair processRegistryUpdates	1.1060 us/op	835.00 ns/op	1.32
mainnet_e81889 - altair processSlashings	220.00 ns/op	403.00 ns/op	0.55
mainnet_e81889 - altair processEth1DataReset	337.00 ns/op	398.00 ns/op	0.85
mainnet_e81889 - altair processEffectiveBalanceUpdates	1.5377 ms/op	1.0300 ms/op	1.49
mainnet_e81889 - altair processSlashingsReset	1.5740 us/op	1.0060 us/op	1.56
mainnet_e81889 - altair processRandaoMixesReset	1.6560 us/op	1.3910 us/op	1.19
mainnet_e81889 - altair processHistoricalRootsUpdate	242.00 ns/op	397.00 ns/op	0.61
mainnet_e81889 - altair processParticipationFlagUpdates	805.00 ns/op	684.00 ns/op	1.18
mainnet_e81889 - altair processSyncCommitteeUpdates	202.00 ns/op	352.00 ns/op	0.57
mainnet_e81889 - altair afterProcessEpoch	65.314 ms/op	42.068 ms/op	1.55
capella processEpoch - mainnet_e217614	1.4323 s/op	1.0302 s/op	1.39
mainnet_e217614 - capella beforeProcessEpoch	96.024 ms/op	69.089 ms/op	1.39
mainnet_e217614 - capella processJustificationAndFinalization	8.2260 us/op	4.7900 us/op	1.72
mainnet_e217614 - capella processInactivityUpdates	24.066 ms/op	14.942 ms/op	1.61
mainnet_e217614 - capella processRewardsAndPenalties	256.52 ms/op	193.97 ms/op	1.32
mainnet_e217614 - capella processRegistryUpdates	8.9170 us/op	6.5870 us/op	1.35
mainnet_e217614 - capella processSlashings	237.00 ns/op	411.00 ns/op	0.58
mainnet_e217614 - capella processEth1DataReset	223.00 ns/op	390.00 ns/op	0.57
mainnet_e217614 - capella processEffectiveBalanceUpdates	18.831 ms/op	17.446 ms/op	1.08
mainnet_e217614 - capella processSlashingsReset	1.1010 us/op	1.1690 us/op	0.94
mainnet_e217614 - capella processRandaoMixesReset	1.7390 us/op	1.5730 us/op	1.11
mainnet_e217614 - capella processHistoricalRootsUpdate	216.00 ns/op	409.00 ns/op	0.53
mainnet_e217614 - capella processParticipationFlagUpdates	723.00 ns/op	792.00 ns/op	0.91
mainnet_e217614 - capella afterProcessEpoch	143.32 ms/op	105.41 ms/op	1.36
phase0 processEpoch - mainnet_e58758	375.18 ms/op	289.71 ms/op	1.30
mainnet_e58758 - phase0 beforeProcessEpoch	98.940 ms/op	78.532 ms/op	1.26
mainnet_e58758 - phase0 processJustificationAndFinalization	7.5750 us/op	5.6660 us/op	1.34
mainnet_e58758 - phase0 processRewardsAndPenalties	42.622 ms/op	40.292 ms/op	1.06
mainnet_e58758 - phase0 processRegistryUpdates	4.4320 us/op	3.3120 us/op	1.34
mainnet_e58758 - phase0 processSlashings	288.00 ns/op	466.00 ns/op	0.62
mainnet_e58758 - phase0 processEth1DataReset	212.00 ns/op	464.00 ns/op	0.46
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.3089 ms/op	819.66 us/op	1.60
mainnet_e58758 - phase0 processSlashingsReset	1.5200 us/op	1.3360 us/op	1.14
mainnet_e58758 - phase0 processRandaoMixesReset	2.1780 us/op	1.4720 us/op	1.48
mainnet_e58758 - phase0 processHistoricalRootsUpdate	223.00 ns/op	424.00 ns/op	0.53
mainnet_e58758 - phase0 processParticipationRecordUpdates	1.1630 us/op	1.2710 us/op	0.92
mainnet_e58758 - phase0 afterProcessEpoch	49.522 ms/op	35.092 ms/op	1.41
phase0 processEffectiveBalanceUpdates - 250000 normalcase	1.9435 ms/op	981.18 us/op	1.98
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	3.2199 ms/op	3.4981 ms/op	0.92
altair processInactivityUpdates - 250000 normalcase	23.163 ms/op	18.304 ms/op	1.27
altair processInactivityUpdates - 250000 worstcase	27.351 ms/op	18.063 ms/op	1.51
phase0 processRegistryUpdates - 250000 normalcase	8.5810 us/op	11.692 us/op	0.73
phase0 processRegistryUpdates - 250000 badcase_full_deposits	353.37 us/op	284.60 us/op	1.24
phase0 processRegistryUpdates - 250000 worstcase 0.5	142.00 ms/op	112.90 ms/op	1.26
altair processRewardsAndPenalties - 250000 normalcase	41.960 ms/op	47.952 ms/op	0.88
altair processRewardsAndPenalties - 250000 worstcase	44.222 ms/op	31.513 ms/op	1.40
phase0 getAttestationDeltas - 250000 normalcase	15.602 ms/op	5.7233 ms/op	2.73
phase0 getAttestationDeltas - 250000 worstcase	18.823 ms/op	5.0277 ms/op	3.74
phase0 processSlashings - 250000 worstcase	126.83 us/op	86.427 us/op	1.47
altair processSyncCommitteeUpdates - 250000	164.12 ms/op	89.817 ms/op	1.83
BeaconState.hashTreeRoot - No change	297.00 ns/op	448.00 ns/op	0.66
BeaconState.hashTreeRoot - 1 full validator	120.18 us/op	73.633 us/op	1.63
BeaconState.hashTreeRoot - 32 full validator	1.5391 ms/op	774.97 us/op	1.99
BeaconState.hashTreeRoot - 512 full validator	15.847 ms/op	7.2376 ms/op	2.19
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	154.78 us/op	86.007 us/op	1.80
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	2.4484 ms/op	1.7408 ms/op	1.41
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	33.382 ms/op	22.525 ms/op	1.48
BeaconState.hashTreeRoot - 1 balances	113.97 us/op	70.695 us/op	1.61
BeaconState.hashTreeRoot - 32 balances	1.2314 ms/op	682.04 us/op	1.81
BeaconState.hashTreeRoot - 512 balances	11.578 ms/op	8.2365 ms/op	1.41
BeaconState.hashTreeRoot - 250000 balances	246.76 ms/op	201.30 ms/op	1.23
aggregationBits - 2048 els - zipIndexesInBitList	29.473 us/op	19.280 us/op	1.53
byteArrayEquals 32	63.006 ns/op	42.719 ns/op	1.47
Buffer.compare 32	20.362 ns/op	17.611 ns/op	1.16
byteArrayEquals 1024	1.9253 us/op	1.2083 us/op	1.59
Buffer.compare 1024	30.037 ns/op	21.483 ns/op	1.40
byteArrayEquals 16384	30.352 us/op	19.759 us/op	1.54
Buffer.compare 16384	215.70 ns/op	171.09 ns/op	1.26
byteArrayEquals 123687377	231.98 ms/op	150.91 ms/op	1.54
Buffer.compare 123687377	9.1315 ms/op	4.6853 ms/op	1.95
byteArrayEquals 32 - diff last byte	58.083 ns/op	47.293 ns/op	1.23
Buffer.compare 32 - diff last byte	19.679 ns/op	16.249 ns/op	1.21
byteArrayEquals 1024 - diff last byte	1.8246 us/op	1.2648 us/op	1.44
Buffer.compare 1024 - diff last byte	30.818 ns/op	24.324 ns/op	1.27
byteArrayEquals 16384 - diff last byte	27.862 us/op	20.059 us/op	1.39
Buffer.compare 16384 - diff last byte	231.48 ns/op	171.51 ns/op	1.35
byteArrayEquals 123687377 - diff last byte	192.34 ms/op	146.94 ms/op	1.31
Buffer.compare 123687377 - diff last byte	6.5756 ms/op	4.6614 ms/op	1.41
byteArrayEquals 32 - random bytes	5.2140 ns/op	5.0090 ns/op	1.04
Buffer.compare 32 - random bytes	17.301 ns/op	15.881 ns/op	1.09
byteArrayEquals 1024 - random bytes	5.1730 ns/op	4.9860 ns/op	1.04
Buffer.compare 1024 - random bytes	17.303 ns/op	15.638 ns/op	1.11
byteArrayEquals 16384 - random bytes	5.1900 ns/op	4.9940 ns/op	1.04
Buffer.compare 16384 - random bytes	17.439 ns/op	15.656 ns/op	1.11
byteArrayEquals 123687377 - random bytes	7.6200 ns/op	7.7100 ns/op	0.99
Buffer.compare 123687377 - random bytes	18.420 ns/op	22.300 ns/op	0.83
regular array get 100000 times	42.159 us/op	31.016 us/op	1.36
wrappedArray get 100000 times	33.595 us/op	31.353 us/op	1.07
arrayWithProxy get 100000 times	12.867 ms/op	10.661 ms/op	1.21
ssz.Root.equals	47.438 ns/op	38.682 ns/op	1.23
byteArrayEquals	46.528 ns/op	41.614 ns/op	1.12
Buffer.compare	10.653 ns/op	8.4740 ns/op	1.26
processSlot - 1 slots	10.707 us/op	10.993 us/op	0.97
processSlot - 32 slots	3.7711 ms/op	2.2872 ms/op	1.65
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	57.109 ms/op	57.997 ms/op	0.98
getCommitteeAssignments - req 1 vs - 250000 vc	2.3165 ms/op	1.8235 ms/op	1.27
getCommitteeAssignments - req 100 vs - 250000 vc	4.2293 ms/op	3.5791 ms/op	1.18
getCommitteeAssignments - req 1000 vs - 250000 vc	4.5808 ms/op	3.8151 ms/op	1.20
findModifiedValidators - 10000 modified validators	776.55 ms/op	765.47 ms/op	1.01
findModifiedValidators - 1000 modified validators	712.42 ms/op	657.69 ms/op	1.08
findModifiedValidators - 100 modified validators	175.02 ms/op	168.32 ms/op	1.04
findModifiedValidators - 10 modified validators	247.63 ms/op	206.03 ms/op	1.20
findModifiedValidators - 1 modified validators	151.07 ms/op	127.54 ms/op	1.18
findModifiedValidators - no difference	180.22 ms/op	155.30 ms/op	1.16
compare ViewDUs	6.6089 s/op	6.0133 s/op	1.10
compare each validator Uint8Array	1.9115 s/op	1.6782 s/op	1.14
compare ViewDU to Uint8Array	1.6062 s/op	810.08 ms/op	1.98
migrate state 1000000 validators, 24 modified, 0 new	897.89 ms/op	841.65 ms/op	1.07
migrate state 1000000 validators, 1700 modified, 1000 new	1.1940 s/op	1.1519 s/op	1.04
migrate state 1000000 validators, 3400 modified, 2000 new	1.4637 s/op	1.2498 s/op	1.17
migrate state 1500000 validators, 24 modified, 0 new	1.0254 s/op	908.61 ms/op	1.13
migrate state 1500000 validators, 1700 modified, 1000 new	1.2735 s/op	1.1980 s/op	1.06
migrate state 1500000 validators, 3400 modified, 2000 new	1.5514 s/op	1.2429 s/op	1.25
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	4.9200 ns/op	6.0500 ns/op	0.81
state getBlockRootAtSlot - 250000 vs - 7PWei	609.38 ns/op	327.75 ns/op	1.86
naive computeProposerIndex 100000 validators	67.079 ms/op	51.921 ms/op	1.29
computeProposerIndex 100000 validators	12.155 ms/op	8.7409 ms/op	1.39
naiveGetNextSyncCommitteeIndices 1000 validators	9.4918 s/op	7.7541 s/op	1.22
getNextSyncCommitteeIndices 1000 validators	348.22 ms/op	233.50 ms/op	1.49
naiveGetNextSyncCommitteeIndices 10000 validators	8.6512 s/op	6.2255 s/op	1.39
getNextSyncCommitteeIndices 10000 validators	280.67 ms/op	196.22 ms/op	1.43
naiveGetNextSyncCommitteeIndices 100000 validators	9.9489 s/op	5.6688 s/op	1.76
getNextSyncCommitteeIndices 100000 validators	502.61 ms/op	210.78 ms/op	2.38
naive computeShuffledIndex 100000 validators	32.295 s/op	18.169 s/op	1.78
cached computeShuffledIndex 100000 validators	686.81 ms/op	477.62 ms/op	1.44
naive computeShuffledIndex 2000000 validators	614.10 s/op	409.11 s/op	1.50
cached computeShuffledIndex 2000000 validators	54.942 s/op	30.692 s/op	1.79
computeProposers - vc 250000	19.144 ms/op	9.2242 ms/op	2.08
computeEpochShuffling - vc 250000	49.560 ms/op	35.641 ms/op	1.39
getNextSyncCommittee - vc 250000	262.70 ms/op	121.42 ms/op	2.16
computeSigningRoot for AttestationData	35.287 us/op	21.838 us/op	1.62
hash AttestationData serialized data then Buffer.toString(base64)	1.9119 us/op	1.1730 us/op	1.63
toHexString serialized data	1.8769 us/op	988.46 ns/op	1.90
Buffer.toString(base64)	197.45 ns/op	103.47 ns/op	1.91
nodejs block root to RootHex using toHex	166.31 ns/op	104.89 ns/op	1.59
nodejs block root to RootHex using toRootHex	142.69 ns/op	68.891 ns/op	2.07
browser block root to RootHex using the deprecated toHexString	260.49 ns/op	175.60 ns/op	1.48
browser block root to RootHex using toHex	194.39 ns/op	151.89 ns/op	1.28
browser block root to RootHex using toRootHex	185.87 ns/op	142.44 ns/op	1.30

by benchmarkbot/action

[twoeths]

verified this on feat1 group

verbose: processSlotsToNearestCheckpoint calls checkpointStateCache to process checkpoint state root=0xf1c7bba5b0507cf18dccfbc2d1f2c29e4c86798a95cffc190fb28968d8248eb8, epoch=115973, persistCount=6

that should help us avoid OOM

[nazarhussain]

Can we not split the logic to pruning cache and processing epoch? Mixing these both here is a bit confusing.

[twoeths]

this processState contains the pruning logic, epoch transition is at line 377 above through processSlots() call

[wemeetagain]

One alternative would be to rely on the ChainEvent.checkpoint event, modifying it to take optionally pass in the checkpoint state.

And then somewhere

onCheckpoint(cp: Checkpoint, checkpointState?: BeaconState) {
  if (!checkpointState) return;
  const latestBlockHex = toRootHex(cp.root);
  try {
    const persistCount = await checkpointStateCache.processState(latestBlockHex, checkpointState);
    logger?.verbose("pruning checkpointStateCache during processSlotsToNearestCheckpoint", {
        root: latestBlockHex,
        epoch: cp.epoch,
        persistCount,
      });
  } catch (e) {
    ...
  }
}

Although the difference is that in this PR, we await the pruning, and ^ we don't await.

[wemeetagain]

LGTM

stale

[wemeetagain]

🎉 This PR is included in v1.28.0 🎉