Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make batch sizes dynamic for eth1 fetch of blocks/logs #4532

Merged
merged 5 commits into from Sep 13, 2022
Merged

Make batch sizes dynamic for eth1 fetch of blocks/logs #4532

merged 5 commits into from Sep 13, 2022

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Sep 10, 2022 •
edited

Motivation
The batched requests of getBlocksByNumber to nethermind would timeout because it seems that the current code flow apparently whips out all 1000 requests in one go via Promise.all

This PR tries to control the concurrency and at any given time doesn't whip out more than ETH1_CONCURRENCY (set to 10) requests to the EL.

Seems to have resolved the problem for user:
https://discord.com/channels/593655374469660673/593655641445367808/1018266744789418055

@github-actions
Copy link
Contributor

github-actions bot commented Sep 10, 2022 •
edited

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 95384ba Previous: 15db5c7 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.2592 ms/op 2.7326 ms/op 0.83
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 59.167 us/op 104.34 us/op 0.57
BLS verify - blst-native 1.8708 ms/op 3.1111 ms/op 0.60
BLS verifyMultipleSignatures 3 - blst-native 3.8635 ms/op 6.8999 ms/op 0.56
BLS verifyMultipleSignatures 8 - blst-native 8.3997 ms/op 15.296 ms/op 0.55
BLS verifyMultipleSignatures 32 - blst-native 30.301 ms/op 51.373 ms/op 0.59
BLS aggregatePubkeys 32 - blst-native 40.793 us/op 70.039 us/op 0.58
BLS aggregatePubkeys 128 - blst-native 157.36 us/op 303.49 us/op 0.52
getAttestationsForBlock 137.56 ms/op 260.78 ms/op 0.53
isKnown best case - 1 super set check 416.00 ns/op 656.00 ns/op 0.63
isKnown normal case - 2 super set checks 403.00 ns/op 636.00 ns/op 0.63
isKnown worse case - 16 super set checks 408.00 ns/op 680.00 ns/op 0.60
CheckpointStateCache - add get delete 7.8090 us/op 14.479 us/op 0.54
validate gossip signedAggregateAndProof - struct 4.3557 ms/op 7.9058 ms/op 0.55
validate gossip attestation - struct 2.0778 ms/op 3.7398 ms/op 0.56
pickEth1Vote - no votes 1.9350 ms/op 3.1671 ms/op 0.61
pickEth1Vote - max votes 17.447 ms/op 30.177 ms/op 0.58
pickEth1Vote - Eth1Data hashTreeRoot value x2048 11.707 ms/op 17.922 ms/op 0.65
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 19.284 ms/op 32.238 ms/op 0.60
pickEth1Vote - Eth1Data fastSerialize value x2048 1.3945 ms/op 2.2356 ms/op 0.62
pickEth1Vote - Eth1Data fastSerialize tree x2048 10.753 ms/op 18.056 ms/op 0.60
bytes32 toHexString 919.00 ns/op 1.5390 us/op 0.60
bytes32 Buffer.toString(hex) 702.00 ns/op 1.0310 us/op 0.68
bytes32 Buffer.toString(hex) from Uint8Array 940.00 ns/op 1.3070 us/op 0.72
bytes32 Buffer.toString(hex) + 0x 709.00 ns/op 1.0150 us/op 0.70
Object access 1 prop 0.34200 ns/op 0.54500 ns/op 0.63
Map access 1 prop 0.27900 ns/op 0.40100 ns/op 0.70
Object get x1000 9.6130 ns/op 20.212 ns/op 0.48
Map get x1000 0.82000 ns/op 1.1690 ns/op 0.70
Object set x1000 66.422 ns/op 147.22 ns/op 0.45
Map set x1000 47.832 ns/op 97.268 ns/op 0.49
Return object 10000 times 0.37400 ns/op 0.49530 ns/op 0.76
Throw Error 10000 times 5.2024 us/op 10.817 us/op 0.48
enrSubnets - fastDeserialize 64 bits 2.4850 us/op 3.8710 us/op 0.64
enrSubnets - ssz BitVector 64 bits 725.00 ns/op 1.0900 us/op 0.67
enrSubnets - fastDeserialize 4 bits 372.00 ns/op 584.00 ns/op 0.64
enrSubnets - ssz BitVector 4 bits 729.00 ns/op 1.1500 us/op 0.63
prioritizePeers score -10:0 att 32-0.1 sync 2-0 77.687 us/op 147.53 us/op 0.53
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 100.36 us/op 225.52 us/op 0.45
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 183.41 us/op 389.79 us/op 0.47
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 356.64 us/op 682.97 us/op 0.52
prioritizePeers score 0:0 att 64-1 sync 4-1 351.53 us/op 837.56 us/op 0.42
RateTracker 1000000 limit, 1 obj count per request 161.68 ns/op 282.32 ns/op 0.57
RateTracker 1000000 limit, 2 obj count per request 121.74 ns/op 196.29 ns/op 0.62
RateTracker 1000000 limit, 4 obj count per request 100.79 ns/op 173.26 ns/op 0.58
RateTracker 1000000 limit, 8 obj count per request 87.908 ns/op 161.77 ns/op 0.54
RateTracker with prune 3.5540 us/op 8.2080 us/op 0.43
array of 16000 items push then shift 44.680 us/op 7.0961 us/op 6.30
LinkedList of 16000 items push then shift 11.259 ns/op 22.731 ns/op 0.50
array of 16000 items push then pop 187.54 ns/op 354.15 ns/op 0.53
LinkedList of 16000 items push then pop 10.573 ns/op 22.948 ns/op 0.46
array of 24000 items push then shift 66.708 us/op 9.3743 us/op 7.12
LinkedList of 24000 items push then shift 11.742 ns/op 24.409 ns/op 0.48
array of 24000 items push then pop 159.31 ns/op 309.94 ns/op 0.51
LinkedList of 24000 items push then pop 10.781 ns/op 21.031 ns/op 0.51
intersect bitArray bitLen 8 9.3750 ns/op 15.257 ns/op 0.61
intersect array and set length 8 124.08 ns/op 245.87 ns/op 0.50
intersect bitArray bitLen 128 49.699 ns/op 86.298 ns/op 0.58
intersect array and set length 128 1.6935 us/op 2.8042 us/op 0.60
Buffer.concat 32 items 1.5760 ns/op 2.6840 ns/op 0.59
pass gossip attestations to forkchoice per slot 4.2661 ms/op 9.0513 ms/op 0.47
computeDeltas 2.9642 ms/op 4.5404 ms/op 0.65
computeProposerBoostScoreFromBalances 700.28 us/op 1.0427 ms/op 0.67
altair processAttestation - 250000 vs - 7PWei normalcase 3.1224 ms/op 6.5060 ms/op 0.48
altair processAttestation - 250000 vs - 7PWei worstcase 5.1176 ms/op 10.605 ms/op 0.48
altair processAttestation - setStatus - 1/6 committees join 163.79 us/op 351.25 us/op 0.47
altair processAttestation - setStatus - 1/3 committees join 318.99 us/op 665.13 us/op 0.48
altair processAttestation - setStatus - 1/2 committees join 450.07 us/op 1.0166 ms/op 0.44
altair processAttestation - setStatus - 2/3 committees join 588.31 us/op 1.1499 ms/op 0.51
altair processAttestation - setStatus - 4/5 committees join 818.22 us/op 1.8738 ms/op 0.44
altair processAttestation - setStatus - 100% committees join 979.01 us/op 2.0492 ms/op 0.48
altair processBlock - 250000 vs - 7PWei normalcase 24.180 ms/op 38.848 ms/op 0.62
altair processBlock - 250000 vs - 7PWei normalcase hashState 34.087 ms/op 50.442 ms/op 0.68
altair processBlock - 250000 vs - 7PWei worstcase 77.967 ms/op 135.57 ms/op 0.58
altair processBlock - 250000 vs - 7PWei worstcase hashState 95.911 ms/op 145.59 ms/op 0.66
phase0 processBlock - 250000 vs - 7PWei normalcase 2.8832 ms/op 5.1143 ms/op 0.56
phase0 processBlock - 250000 vs - 7PWei worstcase 45.674 ms/op 70.625 ms/op 0.65
altair processEth1Data - 250000 vs - 7PWei normalcase 654.29 us/op 1.2953 ms/op 0.51
Tree 40 250000 create 707.90 ms/op 1.3406 s/op 0.53
Tree 40 250000 get(125000) 214.83 ns/op 383.95 ns/op 0.56
Tree 40 250000 set(125000) 2.4536 us/op 5.0109 us/op 0.49
Tree 40 250000 toArray() 29.549 ms/op 49.316 ms/op 0.60
Tree 40 250000 iterate all - toArray() + loop 27.297 ms/op 48.630 ms/op 0.56
Tree 40 250000 iterate all - get(i) 114.58 ms/op 166.50 ms/op 0.69
MutableVector 250000 create 12.639 ms/op 22.032 ms/op 0.57
MutableVector 250000 get(125000) 10.046 ns/op 18.257 ns/op 0.55
MutableVector 250000 set(125000) 460.96 ns/op 1.4360 us/op 0.32
MutableVector 250000 toArray() 5.2810 ms/op 10.613 ms/op 0.50
MutableVector 250000 iterate all - toArray() + loop 5.2689 ms/op 9.6784 ms/op 0.54
MutableVector 250000 iterate all - get(i) 2.2975 ms/op 4.3565 ms/op 0.53
Array 250000 create 5.0798 ms/op 9.8594 ms/op 0.52
Array 250000 clone - spread 2.3997 ms/op 7.7385 ms/op 0.31
Array 250000 get(125000) 1.1730 ns/op 3.2250 ns/op 0.36
Array 250000 set(125000) 1.1670 ns/op 3.5020 ns/op 0.33
Array 250000 iterate all - loop 132.38 us/op 193.11 us/op 0.69
effectiveBalanceIncrements clone Uint8Array 300000 39.112 us/op 513.83 us/op 0.08
effectiveBalanceIncrements clone MutableVector 300000 788.00 ns/op 1.3730 us/op 0.57
effectiveBalanceIncrements rw all Uint8Array 300000 213.39 us/op 394.92 us/op 0.54
effectiveBalanceIncrements rw all MutableVector 300000 142.01 ms/op 299.91 ms/op 0.47
phase0 afterProcessEpoch - 250000 vs - 7PWei 166.84 ms/op 284.91 ms/op 0.59
phase0 beforeProcessEpoch - 250000 vs - 7PWei 87.977 ms/op 99.042 ms/op 0.89
altair processEpoch - mainnet_e81889 537.49 ms/op 722.87 ms/op 0.74
mainnet_e81889 - altair beforeProcessEpoch 130.73 ms/op 201.85 ms/op 0.65
mainnet_e81889 - altair processJustificationAndFinalization 18.131 us/op 67.699 us/op 0.27
mainnet_e81889 - altair processInactivityUpdates 7.9846 ms/op 13.066 ms/op 0.61
mainnet_e81889 - altair processRewardsAndPenalties 82.740 ms/op 114.51 ms/op 0.72
mainnet_e81889 - altair processRegistryUpdates 2.5860 us/op 13.651 us/op 0.19
mainnet_e81889 - altair processSlashings 594.00 ns/op 3.7400 us/op 0.16
mainnet_e81889 - altair processEth1DataReset 648.00 ns/op 4.0090 us/op 0.16
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.6576 ms/op 2.9280 ms/op 0.57
mainnet_e81889 - altair processSlashingsReset 3.9820 us/op 26.158 us/op 0.15
mainnet_e81889 - altair processRandaoMixesReset 3.6480 us/op 25.038 us/op 0.15
mainnet_e81889 - altair processHistoricalRootsUpdate 639.00 ns/op 4.0680 us/op 0.16
mainnet_e81889 - altair processParticipationFlagUpdates 2.1560 us/op 14.693 us/op 0.15
mainnet_e81889 - altair processSyncCommitteeUpdates 509.00 ns/op 3.4590 us/op 0.15
mainnet_e81889 - altair afterProcessEpoch 176.11 ms/op 237.64 ms/op 0.74
phase0 processEpoch - mainnet_e58758 453.24 ms/op 684.68 ms/op 0.66
mainnet_e58758 - phase0 beforeProcessEpoch 191.49 ms/op 322.29 ms/op 0.59
mainnet_e58758 - phase0 processJustificationAndFinalization 17.106 us/op 64.122 us/op 0.27
mainnet_e58758 - phase0 processRewardsAndPenalties 137.33 ms/op 158.67 ms/op 0.87
mainnet_e58758 - phase0 processRegistryUpdates 7.0810 us/op 33.694 us/op 0.21
mainnet_e58758 - phase0 processSlashings 530.00 ns/op 3.1440 us/op 0.17
mainnet_e58758 - phase0 processEth1DataReset 557.00 ns/op 3.4870 us/op 0.16
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.6050 ms/op 3.0116 ms/op 0.53
mainnet_e58758 - phase0 processSlashingsReset 3.0310 us/op 14.931 us/op 0.20
mainnet_e58758 - phase0 processRandaoMixesReset 3.7450 us/op 24.361 us/op 0.15
mainnet_e58758 - phase0 processHistoricalRootsUpdate 572.00 ns/op 4.3270 us/op 0.13
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.2620 us/op 22.398 us/op 0.15
mainnet_e58758 - phase0 afterProcessEpoch 145.05 ms/op 195.59 ms/op 0.74
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.7129 ms/op 3.0587 ms/op 0.56
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.9210 ms/op 3.3545 ms/op 0.57
altair processInactivityUpdates - 250000 normalcase 42.131 ms/op 56.178 ms/op 0.75
altair processInactivityUpdates - 250000 worstcase 41.416 ms/op 74.251 ms/op 0.56
phase0 processRegistryUpdates - 250000 normalcase 5.7190 us/op 34.467 us/op 0.17
phase0 processRegistryUpdates - 250000 badcase_full_deposits 327.45 us/op 625.22 us/op 0.52
phase0 processRegistryUpdates - 250000 worstcase 0.5 174.10 ms/op 321.48 ms/op 0.54
altair processRewardsAndPenalties - 250000 normalcase 117.20 ms/op 174.60 ms/op 0.67
altair processRewardsAndPenalties - 250000 worstcase 77.171 ms/op 107.26 ms/op 0.72
phase0 getAttestationDeltas - 250000 normalcase 11.275 ms/op 18.256 ms/op 0.62
phase0 getAttestationDeltas - 250000 worstcase 11.034 ms/op 17.796 ms/op 0.62
phase0 processSlashings - 250000 worstcase 4.3941 ms/op 7.4585 ms/op 0.59
altair processSyncCommitteeUpdates - 250000 265.56 ms/op 373.93 ms/op 0.71
BeaconState.hashTreeRoot - No change 524.00 ns/op 648.00 ns/op 0.81
BeaconState.hashTreeRoot - 1 full validator 64.453 us/op 93.133 us/op 0.69
BeaconState.hashTreeRoot - 32 full validator 662.55 us/op 815.65 us/op 0.81
BeaconState.hashTreeRoot - 512 full validator 6.0055 ms/op 9.5154 ms/op 0.63
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 82.308 us/op 172.18 us/op 0.48
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.1991 ms/op 1.6012 ms/op 0.75
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 15.783 ms/op 20.820 ms/op 0.76
BeaconState.hashTreeRoot - 1 balances 63.967 us/op 85.666 us/op 0.75
BeaconState.hashTreeRoot - 32 balances 589.49 us/op 809.41 us/op 0.73
BeaconState.hashTreeRoot - 512 balances 7.1226 ms/op 7.7608 ms/op 0.92
BeaconState.hashTreeRoot - 250000 balances 95.920 ms/op 123.34 ms/op 0.78
aggregationBits - 2048 els - zipIndexesInBitList 26.010 us/op 44.333 us/op 0.59
regular array get 100000 times 54.000 us/op 61.366 us/op 0.88
wrappedArray get 100000 times 52.649 us/op 59.208 us/op 0.89
arrayWithProxy get 100000 times 24.471 ms/op 37.509 ms/op 0.65
ssz.Root.equals 434.00 ns/op 670.00 ns/op 0.65
byteArrayEquals 483.00 ns/op 707.00 ns/op 0.68
shuffle list - 16384 els 10.166 ms/op 13.233 ms/op 0.77
shuffle list - 250000 els 147.95 ms/op 194.47 ms/op 0.76
processSlot - 1 slots 12.917 us/op 19.381 us/op 0.67
processSlot - 32 slots 1.8254 ms/op 2.5138 ms/op 0.73
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 364.82 us/op 744.55 us/op 0.49
getCommitteeAssignments - req 1 vs - 250000 vc 4.7208 ms/op 6.0220 ms/op 0.78
getCommitteeAssignments - req 100 vs - 250000 vc 6.8772 ms/op 9.4149 ms/op 0.73
getCommitteeAssignments - req 1000 vs - 250000 vc 7.3187 ms/op 9.2173 ms/op 0.79
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 7.5200 ns/op 10.000 ns/op 0.75
state getBlockRootAtSlot - 250000 vs - 7PWei 999.36 ns/op 1.4393 us/op 0.69
computeProposers - vc 250000 15.891 ms/op 21.697 ms/op 0.73
computeEpochShuffling - vc 250000 151.45 ms/op 197.37 ms/op 0.77
getNextSyncCommittee - vc 250000 262.45 ms/op 366.43 ms/op 0.72

by benchmarkbot/action

@g11tech g11tech marked this pull request as ready for review September 10, 2022 21:18
@g11tech g11tech requested a review from a team as a code owner September 10, 2022 21:18
@dapplion
Copy link
Contributor

Oh damn, that's a fun side effect! To prevent this issue from happening again I think a unit test should be possible. You can create a mock provider and check that the pattern of calls is retried and divided as expected. Let me know if you think that's worth it.

@dapplion dapplion closed this Sep 12, 2022
@dapplion dapplion reopened this Sep 12, 2022
@dapplion
Copy link
Contributor

Sorry hit the wrong button publishing the comment 🙏 ❤️

@g11tech g11tech enabled auto-merge (squash) September 13, 2022 16:28
@g11tech g11tech changed the title Fix getBlocksByNumber batch concurrency Make batch sizes dynamic for eth1 fetch of blocks/logs Sep 13, 2022
@g11tech g11tech enabled auto-merge (squash) September 13, 2022 16:32
wemeetagain
wemeetagain approved these changes Sep 13, 2022
View reviewed changes
Copy link
Member

@wemeetagain wemeetagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

the dynamic fetch is nifty

@g11tech g11tech merged commit 0886d75 into unstable Sep 13, 2022
@g11tech g11tech deleted the g11tech/batched-concurrency branch September 13, 2022 16:55
@g11tech g11tech mentioned this pull request Sep 22, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@g11tech @dapplion @wemeetagain