Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: allow blob archival for bigger time periods #6393

Merged
merged 3 commits into from Feb 16, 2024
Merged

feat: allow blob archival for bigger time periods #6393

merged 3 commits into from Feb 16, 2024

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Feb 5, 2024

a feature requested by EF devops to archive blobs for longer periods

@g11tech g11tech requested a review from a team as a code owner February 5, 2024 17:17
@codecov Codecov
Copy link

codecov bot commented Feb 5, 2024 •
edited

Codecov Report

Merging #6393 (1873034) into unstable (347c95f) will decrease coverage by 0.76%.
Report is 39 commits behind head on unstable.
The diff coverage is n/a.

Additional details and impacted files 
@@             Coverage Diff              @@
##           unstable    #6393      +/-   ##
============================================
- Coverage     60.14%   59.38%   -0.76%     
============================================
  Files           407      388      -19     
  Lines         46485    44802    -1683     
  Branches       1545     1396     -149     
============================================
- Hits          27957    26607    -1350     
+ Misses        18496    18165     -331     
+ Partials         32       30       -2

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024 •
edited

Performance Report

✔️ no performance regression detected

🚀🚀 Significant benchmark improvement detected

Benchmark suite Current: 7c2434f Previous: cb99fc4 Ratio
altair processAttestation - setStatus - 1/6 committees join 67.668 us/op 203.37 us/op 0.33
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 3.4230 us/op 21.139 us/op 0.16
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 800.39 us/op 2.4322 ms/op 0.33
Full benchmark results
Benchmark suite Current: 7c2434f Previous: cb99fc4 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 470.46 us/op 781.72 us/op 0.60
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 109.50 us/op 81.024 us/op 1.35
BLS verify - blst-native 1.1232 ms/op 1.2785 ms/op 0.88
BLS verifyMultipleSignatures 3 - blst-native 2.3980 ms/op 2.6808 ms/op 0.89
BLS verifyMultipleSignatures 8 - blst-native 5.2416 ms/op 5.8822 ms/op 0.89
BLS verifyMultipleSignatures 32 - blst-native 19.426 ms/op 21.498 ms/op 0.90
BLS verifyMultipleSignatures 64 - blst-native 38.358 ms/op 41.692 ms/op 0.92
BLS verifyMultipleSignatures 128 - blst-native 76.103 ms/op 82.488 ms/op 0.92
BLS deserializing 10000 signatures 831.39 ms/op 903.85 ms/op 0.92
BLS deserializing 100000 signatures 8.5099 s/op 9.2095 s/op 0.92
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.1516 ms/op 1.3066 ms/op 0.88
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.2587 ms/op 1.4719 ms/op 0.86
BLS verifyMultipleSignatures - same message - 32 - blst-native 1.9988 ms/op 2.7219 ms/op 0.73
BLS verifyMultipleSignatures - same message - 64 - blst-native 3.7402 ms/op 4.2498 ms/op 0.88
BLS verifyMultipleSignatures - same message - 128 - blst-native 4.9350 ms/op 7.4937 ms/op 0.66
BLS aggregatePubkeys 32 - blst-native 22.472 us/op 27.126 us/op 0.83
BLS aggregatePubkeys 128 - blst-native 87.407 us/op 99.345 us/op 0.88
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 43.502 ms/op 49.096 ms/op 0.89
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 46.602 ms/op 46.645 ms/op 1.00
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 28.512 ms/op 32.538 ms/op 0.88
getSlashingsAndExits - default max 364.32 us/op 216.85 us/op 1.68
getSlashingsAndExits - 2k 312.97 us/op 376.54 us/op 0.83
proposeBlockBody type=full, size=empty 3.8818 ms/op 5.0356 ms/op 0.77
isKnown best case - 1 super set check 327.00 ns/op 296.00 ns/op 1.10
isKnown normal case - 2 super set checks 313.00 ns/op 289.00 ns/op 1.08
isKnown worse case - 16 super set checks 310.00 ns/op 285.00 ns/op 1.09
CheckpointStateCache - add get delete 3.4620 us/op 4.8990 us/op 0.71
validate api signedAggregateAndProof - struct 2.4119 ms/op 2.7149 ms/op 0.89
validate gossip signedAggregateAndProof - struct 2.4309 ms/op 2.7206 ms/op 0.89
validate gossip attestation - vc 640000 1.1344 ms/op 1.3251 ms/op 0.86
batch validate gossip attestation - vc 640000 - chunk 32 136.49 us/op 157.06 us/op 0.87
batch validate gossip attestation - vc 640000 - chunk 64 121.02 us/op 139.37 us/op 0.87
batch validate gossip attestation - vc 640000 - chunk 128 112.29 us/op 134.26 us/op 0.84
batch validate gossip attestation - vc 640000 - chunk 256 107.01 us/op 133.63 us/op 0.80
pickEth1Vote - no votes 809.77 us/op 1.1459 ms/op 0.71
pickEth1Vote - max votes 11.221 ms/op 8.9590 ms/op 1.25
pickEth1Vote - Eth1Data hashTreeRoot value x2048 18.236 ms/op 22.168 ms/op 0.82
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 22.244 ms/op 24.552 ms/op 0.91
pickEth1Vote - Eth1Data fastSerialize value x2048 374.62 us/op 587.73 us/op 0.64
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.5484 ms/op 6.0949 ms/op 1.07
bytes32 toHexString 427.00 ns/op 475.00 ns/op 0.90
bytes32 Buffer.toString(hex) 284.00 ns/op 281.00 ns/op 1.01
bytes32 Buffer.toString(hex) from Uint8Array 393.00 ns/op 406.00 ns/op 0.97
bytes32 Buffer.toString(hex) + 0x 280.00 ns/op 282.00 ns/op 0.99
Object access 1 prop 0.18000 ns/op 0.16100 ns/op 1.12
Map access 1 prop 0.18000 ns/op 0.15000 ns/op 1.20
Object get x1000 4.8250 ns/op 7.0120 ns/op 0.69
Map get x1000 0.66700 ns/op 0.73300 ns/op 0.91
Object set x1000 26.351 ns/op 48.439 ns/op 0.54
Map set x1000 16.673 ns/op 41.436 ns/op 0.40
Return object 10000 times 0.20620 ns/op 0.24350 ns/op 0.85
Throw Error 10000 times 2.6501 us/op 3.8205 us/op 0.69
fastMsgIdFn sha256 / 200 bytes 1.9590 us/op 3.2960 us/op 0.59
fastMsgIdFn h32 xxhash / 200 bytes 281.00 ns/op 286.00 ns/op 0.98
fastMsgIdFn h64 xxhash / 200 bytes 346.00 ns/op 360.00 ns/op 0.96
fastMsgIdFn sha256 / 1000 bytes 6.1050 us/op 11.485 us/op 0.53
fastMsgIdFn h32 xxhash / 1000 bytes 402.00 ns/op 484.00 ns/op 0.83
fastMsgIdFn h64 xxhash / 1000 bytes 413.00 ns/op 460.00 ns/op 0.90
fastMsgIdFn sha256 / 10000 bytes 52.284 us/op 105.14 us/op 0.50
fastMsgIdFn h32 xxhash / 10000 bytes 1.7680 us/op 2.0590 us/op 0.86
fastMsgIdFn h64 xxhash / 10000 bytes 1.2260 us/op 1.4340 us/op 0.85
send data - 1000 256B messages 11.107 ms/op 22.829 ms/op 0.49
send data - 1000 512B messages 13.290 ms/op 28.667 ms/op 0.46
send data - 1000 1024B messages 24.929 ms/op 40.220 ms/op 0.62
send data - 1000 1200B messages 29.551 ms/op 40.766 ms/op 0.72
send data - 1000 2048B messages 37.006 ms/op 53.480 ms/op 0.69
send data - 1000 4096B messages 34.152 ms/op 47.366 ms/op 0.72
send data - 1000 16384B messages 89.395 ms/op 116.28 ms/op 0.77
send data - 1000 65536B messages 360.65 ms/op 441.47 ms/op 0.82
enrSubnets - fastDeserialize 64 bits 1.1970 us/op 1.2340 us/op 0.97
enrSubnets - ssz BitVector 64 bits 437.00 ns/op 404.00 ns/op 1.08
enrSubnets - fastDeserialize 4 bits 189.00 ns/op 170.00 ns/op 1.11
enrSubnets - ssz BitVector 4 bits 420.00 ns/op 414.00 ns/op 1.01
prioritizePeers score -10:0 att 32-0.1 sync 2-0 63.772 us/op 96.828 us/op 0.66
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 82.996 us/op 128.86 us/op 0.64
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 111.20 us/op 159.72 us/op 0.70
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 208.30 us/op 288.64 us/op 0.72
prioritizePeers score 0:0 att 64-1 sync 4-1 260.22 us/op 335.02 us/op 0.78
array of 16000 items push then shift 1.3609 us/op 1.6016 us/op 0.85
LinkedList of 16000 items push then shift 6.3580 ns/op 8.7500 ns/op 0.73
array of 16000 items push then pop 105.67 ns/op 95.453 ns/op 1.11
LinkedList of 16000 items push then pop 5.8850 ns/op 8.5970 ns/op 0.68
array of 24000 items push then shift 1.9924 us/op 2.5319 us/op 0.79
LinkedList of 24000 items push then shift 6.2820 ns/op 8.8390 ns/op 0.71
array of 24000 items push then pop 114.71 ns/op 113.85 ns/op 1.01
LinkedList of 24000 items push then pop 5.7070 ns/op 8.6660 ns/op 0.66
intersect bitArray bitLen 8 4.4260 ns/op 5.8920 ns/op 0.75
intersect array and set length 8 48.604 ns/op 67.392 ns/op 0.72
intersect bitArray bitLen 128 27.803 ns/op 34.590 ns/op 0.80
intersect array and set length 128 649.31 ns/op 836.02 ns/op 0.78
bitArray.getTrueBitIndexes() bitLen 128 1.1990 us/op 1.3860 us/op 0.87
bitArray.getTrueBitIndexes() bitLen 248 1.9670 us/op 2.3620 us/op 0.83
bitArray.getTrueBitIndexes() bitLen 512 3.7670 us/op 4.7030 us/op 0.80
Buffer.concat 32 items 843.00 ns/op 917.00 ns/op 0.92
Uint8Array.set 32 items 2.3220 us/op 1.8880 us/op 1.23
Set add up to 64 items then delete first 1.5878 us/op 4.2949 us/op 0.37
OrderedSet add up to 64 items then delete first 2.4335 us/op 5.3814 us/op 0.45
Set add up to 64 items then delete last 1.8189 us/op 5.0855 us/op 0.36
OrderedSet add up to 64 items then delete last 2.7053 us/op 6.5443 us/op 0.41
Set add up to 64 items then delete middle 1.8024 us/op 4.7267 us/op 0.38
OrderedSet add up to 64 items then delete middle 3.8105 us/op 7.4349 us/op 0.51
Set add up to 128 items then delete first 3.5372 us/op 9.4403 us/op 0.37
OrderedSet add up to 128 items then delete first 5.5273 us/op 12.747 us/op 0.43
Set add up to 128 items then delete last 3.4597 us/op 9.9304 us/op 0.35
OrderedSet add up to 128 items then delete last 5.2944 us/op 11.817 us/op 0.45
Set add up to 128 items then delete middle 3.4386 us/op 9.7361 us/op 0.35
OrderedSet add up to 128 items then delete middle 9.6884 us/op 17.330 us/op 0.56
Set add up to 256 items then delete first 7.1064 us/op 20.382 us/op 0.35
OrderedSet add up to 256 items then delete first 11.560 us/op 25.520 us/op 0.45
Set add up to 256 items then delete last 6.9618 us/op 18.147 us/op 0.38
OrderedSet add up to 256 items then delete last 10.652 us/op 23.422 us/op 0.45
Set add up to 256 items then delete middle 6.9704 us/op 18.422 us/op 0.38
OrderedSet add up to 256 items then delete middle 28.922 us/op 45.030 us/op 0.64
transfer serialized Status (84 B) 1.3660 us/op 1.7060 us/op 0.80
copy serialized Status (84 B) 1.2030 us/op 1.2770 us/op 0.94
transfer serialized SignedVoluntaryExit (112 B) 1.6400 us/op 1.9130 us/op 0.86
copy serialized SignedVoluntaryExit (112 B) 1.3230 us/op 1.4600 us/op 0.91
transfer serialized ProposerSlashing (416 B) 2.6720 us/op 2.9950 us/op 0.89
copy serialized ProposerSlashing (416 B) 2.6680 us/op 2.8340 us/op 0.94
transfer serialized Attestation (485 B) 2.6880 us/op 3.2850 us/op 0.82
copy serialized Attestation (485 B) 2.7150 us/op 2.8780 us/op 0.94
transfer serialized AttesterSlashing (33232 B) 2.7190 us/op 3.2000 us/op 0.85
copy serialized AttesterSlashing (33232 B) 6.2710 us/op 7.1850 us/op 0.87
transfer serialized Small SignedBeaconBlock (128000 B) 3.0420 us/op 3.8180 us/op 0.80
copy serialized Small SignedBeaconBlock (128000 B) 12.412 us/op 15.861 us/op 0.78
transfer serialized Avg SignedBeaconBlock (200000 B) 2.8580 us/op 3.9360 us/op 0.73
copy serialized Avg SignedBeaconBlock (200000 B) 17.926 us/op 24.257 us/op 0.74
transfer serialized BlobsSidecar (524380 B) 2.6480 us/op 3.8080 us/op 0.70
copy serialized BlobsSidecar (524380 B) 67.377 us/op 110.01 us/op 0.61
transfer serialized Big SignedBeaconBlock (1000000 B) 3.1340 us/op 3.8330 us/op 0.82
copy serialized Big SignedBeaconBlock (1000000 B) 127.91 us/op 158.52 us/op 0.81
pass gossip attestations to forkchoice per slot 2.6751 ms/op 4.2971 ms/op 0.62
forkChoice updateHead vc 100000 bc 64 eq 0 458.97 us/op 703.47 us/op 0.65
forkChoice updateHead vc 600000 bc 64 eq 0 2.8068 ms/op 4.1302 ms/op 0.68
forkChoice updateHead vc 1000000 bc 64 eq 0 4.2533 ms/op 6.8351 ms/op 0.62
forkChoice updateHead vc 600000 bc 320 eq 0 2.4562 ms/op 4.0866 ms/op 0.60
forkChoice updateHead vc 600000 bc 1200 eq 0 2.9622 ms/op 4.2527 ms/op 0.70
forkChoice updateHead vc 600000 bc 7200 eq 0 3.2302 ms/op 5.1423 ms/op 0.63
forkChoice updateHead vc 600000 bc 64 eq 1000 10.128 ms/op 10.954 ms/op 0.92
forkChoice updateHead vc 600000 bc 64 eq 10000 10.009 ms/op 11.642 ms/op 0.86
forkChoice updateHead vc 600000 bc 64 eq 300000 12.014 ms/op 15.818 ms/op 0.76
computeDeltas 500000 validators 300 proto nodes 2.9979 ms/op 6.4010 ms/op 0.47
computeDeltas 500000 validators 1200 proto nodes 2.9154 ms/op 6.6543 ms/op 0.44
computeDeltas 500000 validators 7200 proto nodes 2.8934 ms/op 6.5841 ms/op 0.44
computeDeltas 750000 validators 300 proto nodes 4.5543 ms/op 10.381 ms/op 0.44
computeDeltas 750000 validators 1200 proto nodes 4.5396 ms/op 11.263 ms/op 0.40
computeDeltas 750000 validators 7200 proto nodes 4.6071 ms/op 10.876 ms/op 0.42
computeDeltas 1400000 validators 300 proto nodes 8.6137 ms/op 21.317 ms/op 0.40
computeDeltas 1400000 validators 1200 proto nodes 8.8744 ms/op 20.108 ms/op 0.44
computeDeltas 1400000 validators 7200 proto nodes 8.9310 ms/op 20.248 ms/op 0.44
computeDeltas 2100000 validators 300 proto nodes 13.541 ms/op 29.790 ms/op 0.45
computeDeltas 2100000 validators 1200 proto nodes 13.169 ms/op 30.143 ms/op 0.44
computeDeltas 2100000 validators 7200 proto nodes 13.535 ms/op 29.880 ms/op 0.45
altair processAttestation - 250000 vs - 7PWei normalcase 1.4985 ms/op 3.1623 ms/op 0.47
altair processAttestation - 250000 vs - 7PWei worstcase 2.2352 ms/op 4.6440 ms/op 0.48
altair processAttestation - setStatus - 1/6 committees join 67.668 us/op 203.37 us/op 0.33
altair processAttestation - setStatus - 1/3 committees join 143.54 us/op 382.98 us/op 0.37
altair processAttestation - setStatus - 1/2 committees join 206.05 us/op 516.15 us/op 0.40
altair processAttestation - setStatus - 2/3 committees join 260.45 us/op 636.69 us/op 0.41
altair processAttestation - setStatus - 4/5 committees join 388.15 us/op 959.94 us/op 0.40
altair processAttestation - setStatus - 100% committees join 472.31 us/op 1.1557 ms/op 0.41
altair processBlock - 250000 vs - 7PWei normalcase 8.2864 ms/op 13.730 ms/op 0.60
altair processBlock - 250000 vs - 7PWei normalcase hashState 33.818 ms/op 42.443 ms/op 0.80
altair processBlock - 250000 vs - 7PWei worstcase 32.612 ms/op 41.982 ms/op 0.78
altair processBlock - 250000 vs - 7PWei worstcase hashState 77.973 ms/op 104.01 ms/op 0.75
phase0 processBlock - 250000 vs - 7PWei normalcase 2.3056 ms/op 3.0790 ms/op 0.75
phase0 processBlock - 250000 vs - 7PWei worstcase 23.503 ms/op 33.161 ms/op 0.71
altair processEth1Data - 250000 vs - 7PWei normalcase 281.99 us/op 735.01 us/op 0.38
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 3.4230 us/op 21.139 us/op 0.16
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 66.739 us/op 73.998 us/op 0.90
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 8.7220 us/op 23.975 us/op 0.36
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 11.164 us/op 12.388 us/op 0.90
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 147.37 us/op 238.39 us/op 0.62
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.0517 ms/op 2.0913 ms/op 0.50
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 800.39 us/op 2.4322 ms/op 0.33
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.4294 ms/op 2.3467 ms/op 0.61
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 1.9543 ms/op 5.2909 ms/op 0.37
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.5361 ms/op 2.6495 ms/op 0.58
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 3.4715 ms/op 6.5981 ms/op 0.53
Tree 40 250000 create 275.94 ms/op 535.46 ms/op 0.52
Tree 40 250000 get(125000) 100.73 ns/op 228.22 ns/op 0.44
Tree 40 250000 set(125000) 748.94 ns/op 1.4383 us/op 0.52
Tree 40 250000 toArray() 14.612 ms/op 24.810 ms/op 0.59
Tree 40 250000 iterate all - toArray() + loop 11.012 ms/op 25.252 ms/op 0.44
Tree 40 250000 iterate all - get(i) 38.245 ms/op 78.896 ms/op 0.48
MutableVector 250000 create 9.0499 ms/op 15.406 ms/op 0.59
MutableVector 250000 get(125000) 5.6360 ns/op 7.6170 ns/op 0.74
MutableVector 250000 set(125000) 203.47 ns/op 590.63 ns/op 0.34
MutableVector 250000 toArray() 2.0501 ms/op 4.9283 ms/op 0.42
MutableVector 250000 iterate all - toArray() + loop 2.1725 ms/op 4.7994 ms/op 0.45
MutableVector 250000 iterate all - get(i) 1.3602 ms/op 1.7937 ms/op 0.76
Array 250000 create 1.9455 ms/op 4.3540 ms/op 0.45
Array 250000 clone - spread 1.1714 ms/op 1.5474 ms/op 0.76
Array 250000 get(125000) 1.0730 ns/op 1.7390 ns/op 0.62
Array 250000 set(125000) 1.2840 ns/op 5.3690 ns/op 0.24
Array 250000 iterate all - loop 150.43 us/op 180.71 us/op 0.83
effectiveBalanceIncrements clone Uint8Array 300000 20.525 us/op 48.579 us/op 0.42
effectiveBalanceIncrements clone MutableVector 300000 410.00 ns/op 396.00 ns/op 1.04
effectiveBalanceIncrements rw all Uint8Array 300000 177.30 us/op 206.95 us/op 0.86
effectiveBalanceIncrements rw all MutableVector 300000 65.476 ms/op 103.46 ms/op 0.63
phase0 afterProcessEpoch - 250000 vs - 7PWei 75.455 ms/op 127.33 ms/op 0.59
phase0 beforeProcessEpoch - 250000 vs - 7PWei 42.846 ms/op 66.226 ms/op 0.65
altair processEpoch - mainnet_e81889 400.09 ms/op 559.10 ms/op 0.72
mainnet_e81889 - altair beforeProcessEpoch 68.166 ms/op 93.815 ms/op 0.73
mainnet_e81889 - altair processJustificationAndFinalization 9.1880 us/op 17.811 us/op 0.52
mainnet_e81889 - altair processInactivityUpdates 3.7701 ms/op 6.6234 ms/op 0.57
mainnet_e81889 - altair processRewardsAndPenalties 50.421 ms/op 42.611 ms/op 1.18
mainnet_e81889 - altair processRegistryUpdates 1.9610 us/op 2.9660 us/op 0.66
mainnet_e81889 - altair processSlashings 647.00 ns/op 759.00 ns/op 0.85
mainnet_e81889 - altair processEth1DataReset 635.00 ns/op 1.8300 us/op 0.35
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2319 ms/op 1.4639 ms/op 0.84
mainnet_e81889 - altair processSlashingsReset 1.3650 us/op 7.9900 us/op 0.17
mainnet_e81889 - altair processRandaoMixesReset 2.5590 us/op 7.9420 us/op 0.32
mainnet_e81889 - altair processHistoricalRootsUpdate 647.00 ns/op 1.5950 us/op 0.41
mainnet_e81889 - altair processParticipationFlagUpdates 1.6110 us/op 6.3540 us/op 0.25
mainnet_e81889 - altair processSyncCommitteeUpdates 518.00 ns/op 2.0070 us/op 0.26
mainnet_e81889 - altair afterProcessEpoch 76.778 ms/op 124.24 ms/op 0.62
capella processEpoch - mainnet_e217614 1.7294 s/op 2.5352 s/op 0.68
mainnet_e217614 - capella beforeProcessEpoch 415.72 ms/op 486.06 ms/op 0.86
mainnet_e217614 - capella processJustificationAndFinalization 10.135 us/op 19.133 us/op 0.53
mainnet_e217614 - capella processInactivityUpdates 14.945 ms/op 21.198 ms/op 0.71
mainnet_e217614 - capella processRewardsAndPenalties 358.88 ms/op 422.81 ms/op 0.85
mainnet_e217614 - capella processRegistryUpdates 16.087 us/op 17.112 us/op 0.94
mainnet_e217614 - capella processSlashings 584.00 ns/op 578.00 ns/op 1.01
mainnet_e217614 - capella processEth1DataReset 514.00 ns/op 452.00 ns/op 1.14
mainnet_e217614 - capella processEffectiveBalanceUpdates 3.8247 ms/op 5.6264 ms/op 0.68
mainnet_e217614 - capella processSlashingsReset 2.0140 us/op 3.6630 us/op 0.55
mainnet_e217614 - capella processRandaoMixesReset 3.5600 us/op 4.5600 us/op 0.78
mainnet_e217614 - capella processHistoricalRootsUpdate 1.0410 us/op 427.00 ns/op 2.44
mainnet_e217614 - capella processParticipationFlagUpdates 1.1120 us/op 1.4300 us/op 0.78
mainnet_e217614 - capella afterProcessEpoch 206.75 ms/op 301.82 ms/op 0.69
phase0 processEpoch - mainnet_e58758 428.14 ms/op 436.38 ms/op 0.98
mainnet_e58758 - phase0 beforeProcessEpoch 122.73 ms/op 125.38 ms/op 0.98
mainnet_e58758 - phase0 processJustificationAndFinalization 12.778 us/op 17.257 us/op 0.74
mainnet_e58758 - phase0 processRewardsAndPenalties 46.685 ms/op 37.873 ms/op 1.23
mainnet_e58758 - phase0 processRegistryUpdates 4.1070 us/op 11.921 us/op 0.34
mainnet_e58758 - phase0 processSlashings 330.00 ns/op 420.00 ns/op 0.79
mainnet_e58758 - phase0 processEth1DataReset 314.00 ns/op 426.00 ns/op 0.74
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 847.67 us/op 1.1714 ms/op 0.72
mainnet_e58758 - phase0 processSlashingsReset 2.0500 us/op 3.0230 us/op 0.68
mainnet_e58758 - phase0 processRandaoMixesReset 2.6010 us/op 5.8810 us/op 0.44
mainnet_e58758 - phase0 processHistoricalRootsUpdate 577.00 ns/op 543.00 ns/op 1.06
mainnet_e58758 - phase0 processParticipationRecordUpdates 2.8710 us/op 4.4510 us/op 0.65
mainnet_e58758 - phase0 afterProcessEpoch 61.776 ms/op 97.974 ms/op 0.63
phase0 processEffectiveBalanceUpdates - 250000 normalcase 993.04 us/op 1.3860 ms/op 0.72
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.1494 ms/op 1.9794 ms/op 0.58
altair processInactivityUpdates - 250000 normalcase 21.335 ms/op 23.606 ms/op 0.90
altair processInactivityUpdates - 250000 worstcase 21.727 ms/op 20.158 ms/op 1.08
phase0 processRegistryUpdates - 250000 normalcase 5.9280 us/op 8.1010 us/op 0.73
phase0 processRegistryUpdates - 250000 badcase_full_deposits 253.06 us/op 410.16 us/op 0.62
phase0 processRegistryUpdates - 250000 worstcase 0.5 113.38 ms/op 125.88 ms/op 0.90
altair processRewardsAndPenalties - 250000 normalcase 56.129 ms/op 42.949 ms/op 1.31
altair processRewardsAndPenalties - 250000 worstcase 49.870 ms/op 40.880 ms/op 1.22
phase0 getAttestationDeltas - 250000 normalcase 6.3123 ms/op 8.9342 ms/op 0.71
phase0 getAttestationDeltas - 250000 worstcase 5.7969 ms/op 9.0759 ms/op 0.64
phase0 processSlashings - 250000 worstcase 81.412 us/op 82.023 us/op 0.99
altair processSyncCommitteeUpdates - 250000 105.66 ms/op 155.67 ms/op 0.68
BeaconState.hashTreeRoot - No change 356.00 ns/op 376.00 ns/op 0.95
BeaconState.hashTreeRoot - 1 full validator 128.49 us/op 122.80 us/op 1.05
BeaconState.hashTreeRoot - 32 full validator 1.4143 ms/op 1.5146 ms/op 0.93
BeaconState.hashTreeRoot - 512 full validator 14.377 ms/op 13.752 ms/op 1.05
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 188.22 us/op 145.10 us/op 1.30
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.8736 ms/op 2.1634 ms/op 0.87
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 26.232 ms/op 23.908 ms/op 1.10
BeaconState.hashTreeRoot - 1 balances 103.16 us/op 107.98 us/op 0.96
BeaconState.hashTreeRoot - 32 balances 931.83 us/op 1.2388 ms/op 0.75
BeaconState.hashTreeRoot - 512 balances 7.6935 ms/op 13.398 ms/op 0.57
BeaconState.hashTreeRoot - 250000 balances 142.39 ms/op 229.58 ms/op 0.62
aggregationBits - 2048 els - zipIndexesInBitList 13.019 us/op 18.553 us/op 0.70
byteArrayEquals 32 61.181 ns/op 75.711 ns/op 0.81
Buffer.compare 32 34.676 ns/op 56.832 ns/op 0.61
byteArrayEquals 1024 1.6210 us/op 2.0780 us/op 0.78
Buffer.compare 1024 40.075 ns/op 71.998 ns/op 0.56
byteArrayEquals 16384 27.633 us/op 32.686 us/op 0.85
Buffer.compare 16384 232.08 ns/op 244.73 ns/op 0.95
byteArrayEquals 123687377 198.09 ms/op 263.96 ms/op 0.75
Buffer.compare 123687377 5.9811 ms/op 8.9139 ms/op 0.67
byteArrayEquals 32 - diff last byte 60.833 ns/op 84.093 ns/op 0.72
Buffer.compare 32 - diff last byte 37.187 ns/op 58.951 ns/op 0.63
byteArrayEquals 1024 - diff last byte 1.6720 us/op 2.2338 us/op 0.75
Buffer.compare 1024 - diff last byte 47.148 ns/op 78.283 ns/op 0.60
byteArrayEquals 16384 - diff last byte 27.973 us/op 34.887 us/op 0.80
Buffer.compare 16384 - diff last byte 234.91 ns/op 301.11 ns/op 0.78
byteArrayEquals 123687377 - diff last byte 199.16 ms/op 272.26 ms/op 0.73
Buffer.compare 123687377 - diff last byte 5.6379 ms/op 9.7619 ms/op 0.58
byteArrayEquals 32 - random bytes 4.5470 ns/op 6.2760 ns/op 0.72
Buffer.compare 32 - random bytes 37.228 ns/op 63.914 ns/op 0.58
byteArrayEquals 1024 - random bytes 4.3760 ns/op 7.1700 ns/op 0.61
Buffer.compare 1024 - random bytes 34.873 ns/op 63.377 ns/op 0.55
byteArrayEquals 16384 - random bytes 4.4290 ns/op 6.3050 ns/op 0.70
Buffer.compare 16384 - random bytes 34.956 ns/op 63.362 ns/op 0.55
byteArrayEquals 123687377 - random bytes 7.7300 ns/op 9.2900 ns/op 0.83
Buffer.compare 123687377 - random bytes 37.280 ns/op 76.700 ns/op 0.49
regular array get 100000 times 39.800 us/op 46.504 us/op 0.86
wrappedArray get 100000 times 39.782 us/op 46.481 us/op 0.86
arrayWithProxy get 100000 times 9.6433 ms/op 14.805 ms/op 0.65
ssz.Root.equals 55.430 ns/op 58.481 ns/op 0.95
byteArrayEquals 53.238 ns/op 57.281 ns/op 0.93
Buffer.compare 8.8570 ns/op 12.741 ns/op 0.70
shuffle list - 16384 els 4.5948 ms/op 7.3652 ms/op 0.62
shuffle list - 250000 els 70.222 ms/op 106.96 ms/op 0.66
processSlot - 1 slots 17.375 us/op 18.632 us/op 0.93
processSlot - 32 slots 3.2599 ms/op 3.2672 ms/op 1.00
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 46.123 ms/op 61.870 ms/op 0.75
getCommitteeAssignments - req 1 vs - 250000 vc 2.2567 ms/op 2.5147 ms/op 0.90
getCommitteeAssignments - req 100 vs - 250000 vc 3.3623 ms/op 3.7257 ms/op 0.90
getCommitteeAssignments - req 1000 vs - 250000 vc 3.7102 ms/op 4.0430 ms/op 0.92
findModifiedValidators - 10000 modified validators 416.79 ms/op 531.88 ms/op 0.78
findModifiedValidators - 1000 modified validators 369.35 ms/op 467.55 ms/op 0.79
findModifiedValidators - 100 modified validators 329.03 ms/op 464.25 ms/op 0.71
findModifiedValidators - 10 modified validators 352.53 ms/op 427.86 ms/op 0.82
findModifiedValidators - 1 modified validators 353.84 ms/op 425.03 ms/op 0.83
findModifiedValidators - no difference 332.83 ms/op 420.26 ms/op 0.79
compare ViewDUs 4.0023 s/op 4.8502 s/op 0.83
compare each validator Uint8Array 1.2981 s/op 2.0767 s/op 0.63
compare ViewDU to Uint8Array 782.59 ms/op 1.4589 s/op 0.54
migrate state 1000000 validators, 24 modified, 0 new 617.76 ms/op 881.48 ms/op 0.70
migrate state 1000000 validators, 1700 modified, 1000 new 1.0038 s/op 1.0784 s/op 0.93
migrate state 1000000 validators, 3400 modified, 2000 new 1.2778 s/op 1.3290 s/op 0.96
migrate state 1500000 validators, 24 modified, 0 new 728.64 ms/op 770.13 ms/op 0.95
migrate state 1500000 validators, 1700 modified, 1000 new 993.55 ms/op 1.1069 s/op 0.90
migrate state 1500000 validators, 3400 modified, 2000 new 1.1249 s/op 1.2666 s/op 0.89
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.3900 ns/op 4.2500 ns/op 1.03
state getBlockRootAtSlot - 250000 vs - 7PWei 393.89 ns/op 849.71 ns/op 0.46
computeProposers - vc 250000 5.8203 ms/op 8.8213 ms/op 0.66
computeEpochShuffling - vc 250000 70.676 ms/op 103.30 ms/op 0.68
getNextSyncCommittee - vc 250000 99.273 ms/op 147.99 ms/op 0.67
computeSigningRoot for AttestationData 22.606 us/op 25.285 us/op 0.89
hash AttestationData serialized data then Buffer.toString(base64) 1.2137 us/op 2.3199 us/op 0.52
toHexString serialized data 790.27 ns/op 1.1311 us/op 0.70
Buffer.toString(base64) 161.13 ns/op 227.78 ns/op 0.71

by benchmarkbot/action

@philknows philknows added this to the v1.16.0 milestone Feb 9, 2024
matthewkeil
matthewkeil approved these changes Feb 15, 2024
View reviewed changes
Copy link
Member

@matthewkeil matthewkeil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense! LGTM!! 🚀

Would anyone ever want to add a way to prune the DB on restart if the consumer wants the flag to go away? I suppose just deleting the db is fine but figured it was worth asking just in case.

nflaig
nflaig reviewed Feb 15, 2024
View reviewed changes
packages/beacon-node/src/chain/archiver/archiveBlocks.ts Show resolved Hide resolved
packages/beacon-node/src/chain/archiver/archiveBlocks.ts
logger.verbose(`blobSidecars prune: no entries before epoch ${blobSidecarsMinEpoch}`);
if (archiveBlobEpochs !== Infinity) {
const blobsArchiveWindow = Math.max(config.MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS, archiveBlobEpochs ?? 0);
const blobSidecarsMinEpoch = currentEpoch - blobsArchiveWindow;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we are fine with this being a negative number, in that case, do we even need to handle Infinity explicitly?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thats a good point. Would just get skipped as normal, although the log output would be funny...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

although the log output would be funny

It is not that bad

blobSidecars prune: no entries before epoch -Infinity

but I guess the other log is more explicit about it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its good to be explicit not just for code correctness but also because for code readability and understanding which is one of the lodestar's objective (to stay readable and declarative)

@g11tech g11tech merged commit 6a68b1e into unstable Feb 16, 2024
14 of 15 checks passed
@g11tech g11tech deleted the archive-blobs branch February 16, 2024 15:07
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.16.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain left review comments

@matthewkeil matthewkeil matthewkeil approved these changes

@nflaig nflaig nflaig left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v1.16.0
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@g11tech @wemeetagain @matthewkeil @nflaig @philknows