Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable workers for e2e test runs #4325

Merged
merged 1 commit into from Jul 18, 2022

Conversation

dapplion
Copy link
Contributor

Motivation

Description

  • Disable workers for e2e test runs

We are running way too many Lodestar nodes in e2e tests

@dapplion dapplion requested a review from a team as a code owner July 18, 2022 17:02
@github-actions
Copy link
Contributor

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite Current: 8aa3f51 Previous: fdd4c6b Ratio
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 1.4798 ms/op 382.59 us/op 3.87
Full benchmark results
Benchmark suite Current: 8aa3f51 Previous: fdd4c6b Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.5328 ms/op 1.7909 ms/op 1.41
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 83.936 us/op 64.738 us/op 1.30
BLS verify - blst-native 2.2237 ms/op 2.1673 ms/op 1.03
BLS verifyMultipleSignatures 3 - blst-native 4.5095 ms/op 4.4711 ms/op 1.01
BLS verifyMultipleSignatures 8 - blst-native 9.6149 ms/op 9.6702 ms/op 0.99
BLS verifyMultipleSignatures 32 - blst-native 34.779 ms/op 35.132 ms/op 0.99
BLS aggregatePubkeys 32 - blst-native 46.579 us/op 46.815 us/op 0.99
BLS aggregatePubkeys 128 - blst-native 182.54 us/op 182.70 us/op 1.00
getAttestationsForBlock 55.307 ms/op 41.850 ms/op 1.32
isKnown best case - 1 super set check 514.00 ns/op 488.00 ns/op 1.05
isKnown normal case - 2 super set checks 503.00 ns/op 475.00 ns/op 1.06
isKnown worse case - 16 super set checks 497.00 ns/op 478.00 ns/op 1.04
CheckpointStateCache - add get delete 10.469 us/op 9.2340 us/op 1.13
validate gossip signedAggregateAndProof - struct 5.0806 ms/op 5.0270 ms/op 1.01
validate gossip attestation - struct 2.3838 ms/op 2.3751 ms/op 1.00
altair verifyImport mainnet_s3766816:31 13.926 s/op 12.664 s/op 1.10
pickEth1Vote - no votes 2.4761 ms/op 2.1150 ms/op 1.17
pickEth1Vote - max votes 27.452 ms/op 20.230 ms/op 1.36
pickEth1Vote - Eth1Data hashTreeRoot value x2048 13.621 ms/op 13.317 ms/op 1.02
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 25.676 ms/op 20.891 ms/op 1.23
pickEth1Vote - Eth1Data fastSerialize value x2048 1.8844 ms/op 1.4652 ms/op 1.29
pickEth1Vote - Eth1Data fastSerialize tree x2048 18.839 ms/op 14.605 ms/op 1.29
bytes32 toHexString 1.3450 us/op 1.0530 us/op 1.28
bytes32 Buffer.toString(hex) 889.00 ns/op 788.00 ns/op 1.13
bytes32 Buffer.toString(hex) from Uint8Array 1.1150 us/op 1.0140 us/op 1.10
bytes32 Buffer.toString(hex) + 0x 899.00 ns/op 798.00 ns/op 1.13
Object access 1 prop 0.46900 ns/op 0.38200 ns/op 1.23
Map access 1 prop 0.33400 ns/op 0.30300 ns/op 1.10
Object get x1000 20.982 ns/op 11.373 ns/op 1.84
Map get x1000 1.1460 ns/op 0.95500 ns/op 1.20
Object set x1000 145.41 ns/op 75.630 ns/op 1.92
Map set x1000 90.858 ns/op 49.283 ns/op 1.84
Return object 10000 times 0.44570 ns/op 0.44360 ns/op 1.00
Throw Error 10000 times 7.0810 us/op 6.0727 us/op 1.17
enrSubnets - fastDeserialize 64 bits 3.4160 us/op 2.8210 us/op 1.21
enrSubnets - ssz BitVector 64 bits 878.00 ns/op 789.00 ns/op 1.11
enrSubnets - fastDeserialize 4 bits 485.00 ns/op 393.00 ns/op 1.23
enrSubnets - ssz BitVector 4 bits 863.00 ns/op 753.00 ns/op 1.15
prioritizePeers score -10:0 att 32-0.1 sync 2-0 117.28 us/op 87.711 us/op 1.34
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 160.49 us/op 115.63 us/op 1.39
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 280.16 us/op 211.91 us/op 1.32
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 438.40 us/op 410.74 us/op 1.07
prioritizePeers score 0:0 att 64-1 sync 4-1 537.09 us/op 407.31 us/op 1.32
RateTracker 1000000 limit, 1 obj count per request 228.97 ns/op 205.97 ns/op 1.11
RateTracker 1000000 limit, 2 obj count per request 172.25 ns/op 151.92 ns/op 1.13
RateTracker 1000000 limit, 4 obj count per request 146.51 ns/op 122.73 ns/op 1.19
RateTracker 1000000 limit, 8 obj count per request 132.07 ns/op 105.59 ns/op 1.25
RateTracker with prune 5.2500 us/op 4.3930 us/op 1.20
array of 16000 items push then shift 3.7764 us/op 51.583 us/op 0.07
LinkedList of 16000 items push then shift 29.352 ns/op 15.700 ns/op 1.87
array of 16000 items push then pop 279.17 ns/op 236.92 ns/op 1.18
LinkedList of 16000 items push then pop 23.203 ns/op 14.247 ns/op 1.63
array of 24000 items push then shift 5.4900 us/op 77.340 us/op 0.07
LinkedList of 24000 items push then shift 31.317 ns/op 19.377 ns/op 1.62
array of 24000 items push then pop 239.61 ns/op 195.20 ns/op 1.23
LinkedList of 24000 items push then pop 23.940 ns/op 15.871 ns/op 1.51
intersect bitArray bitLen 8 13.495 ns/op 10.662 ns/op 1.27
intersect array and set length 8 198.12 ns/op 147.58 ns/op 1.34
intersect bitArray bitLen 128 71.797 ns/op 57.891 ns/op 1.24
intersect array and set length 128 2.7670 us/op 1.9582 us/op 1.41
pass gossip attestations to forkchoice per slot 4.1547 ms/op 3.1778 ms/op 1.31
computeDeltas 3.7545 ms/op 3.2307 ms/op 1.16
computeProposerBoostScoreFromBalances 1.0679 ms/op 809.15 us/op 1.32
altair processAttestation - 250000 vs - 7PWei normalcase 4.9216 ms/op 3.7314 ms/op 1.32
altair processAttestation - 250000 vs - 7PWei worstcase 7.2761 ms/op 5.4880 ms/op 1.33
altair processAttestation - setStatus - 1/6 committees join 272.08 us/op 188.50 us/op 1.44
altair processAttestation - setStatus - 1/3 committees join 499.13 us/op 366.12 us/op 1.36
altair processAttestation - setStatus - 1/2 committees join 746.74 us/op 525.94 us/op 1.42
altair processAttestation - setStatus - 2/3 committees join 914.35 us/op 694.14 us/op 1.32
altair processAttestation - setStatus - 4/5 committees join 1.3255 ms/op 953.22 us/op 1.39
altair processAttestation - setStatus - 100% committees join 1.5045 ms/op 1.1569 ms/op 1.30
altair processBlock - 250000 vs - 7PWei normalcase 31.715 ms/op 27.696 ms/op 1.15
altair processBlock - 250000 vs - 7PWei normalcase hashState 45.694 ms/op 33.378 ms/op 1.37
altair processBlock - 250000 vs - 7PWei worstcase 106.83 ms/op 83.414 ms/op 1.28
altair processBlock - 250000 vs - 7PWei worstcase hashState 115.84 ms/op 101.38 ms/op 1.14
phase0 processBlock - 250000 vs - 7PWei normalcase 5.6348 ms/op 4.2065 ms/op 1.34
phase0 processBlock - 250000 vs - 7PWei worstcase 57.353 ms/op 53.067 ms/op 1.08
altair processEth1Data - 250000 vs - 7PWei normalcase 1.0674 ms/op 772.82 us/op 1.38
Tree 40 250000 create 904.50 ms/op 762.78 ms/op 1.19
Tree 40 250000 get(125000) 345.69 ns/op 251.43 ns/op 1.37
Tree 40 250000 set(125000) 2.8584 us/op 2.6116 us/op 1.09
Tree 40 250000 toArray() 38.193 ms/op 28.912 ms/op 1.32
Tree 40 250000 iterate all - toArray() + loop 37.231 ms/op 29.976 ms/op 1.24
Tree 40 250000 iterate all - get(i) 129.41 ms/op 110.78 ms/op 1.17
MutableVector 250000 create 19.383 ms/op 12.439 ms/op 1.56
MutableVector 250000 get(125000) 15.405 ns/op 10.770 ns/op 1.43
MutableVector 250000 set(125000) 744.97 ns/op 472.55 ns/op 1.58
MutableVector 250000 toArray() 8.7337 ms/op 5.8123 ms/op 1.50
MutableVector 250000 iterate all - toArray() + loop 8.8012 ms/op 6.0507 ms/op 1.45
MutableVector 250000 iterate all - get(i) 4.1242 ms/op 2.6775 ms/op 1.54
Array 250000 create 8.0988 ms/op 5.5431 ms/op 1.46
Array 250000 clone - spread 4.9384 ms/op 2.5564 ms/op 1.93
Array 250000 get(125000) 2.1530 ns/op 1.2350 ns/op 1.74
Array 250000 set(125000) 2.0300 ns/op 1.2300 ns/op 1.65
Array 250000 iterate all - loop 200.36 us/op 151.27 us/op 1.32
effectiveBalanceIncrements clone Uint8Array 300000 117.78 us/op 76.720 us/op 1.54
effectiveBalanceIncrements clone MutableVector 300000 1.5760 us/op 611.00 ns/op 2.58
effectiveBalanceIncrements rw all Uint8Array 300000 303.49 us/op 247.55 us/op 1.23
effectiveBalanceIncrements rw all MutableVector 300000 250.29 ms/op 134.05 ms/op 1.87
phase0 afterProcessEpoch - 250000 vs - 7PWei 240.98 ms/op 186.56 ms/op 1.29
phase0 beforeProcessEpoch - 250000 vs - 7PWei 108.32 ms/op 58.308 ms/op 1.86
altair processEpoch - mainnet_e81889 686.46 ms/op 549.13 ms/op 1.25
mainnet_e81889 - altair beforeProcessEpoch 181.82 ms/op 116.93 ms/op 1.55
mainnet_e81889 - altair processJustificationAndFinalization 38.120 us/op 16.296 us/op 2.34
mainnet_e81889 - altair processInactivityUpdates 12.527 ms/op 8.8829 ms/op 1.41
mainnet_e81889 - altair processRewardsAndPenalties 110.77 ms/op 82.020 ms/op 1.35
mainnet_e81889 - altair processRegistryUpdates 9.3350 us/op 2.6620 us/op 3.51
mainnet_e81889 - altair processSlashings 2.1940 us/op 676.00 ns/op 3.25
mainnet_e81889 - altair processEth1DataReset 2.1490 us/op 654.00 ns/op 3.29
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.8660 ms/op 1.9596 ms/op 1.46
mainnet_e81889 - altair processSlashingsReset 11.124 us/op 4.2360 us/op 2.63
mainnet_e81889 - altair processRandaoMixesReset 12.856 us/op 3.9440 us/op 3.26
mainnet_e81889 - altair processHistoricalRootsUpdate 2.2940 us/op 588.00 ns/op 3.90
mainnet_e81889 - altair processParticipationFlagUpdates 6.7930 us/op 3.0620 us/op 2.22
mainnet_e81889 - altair processSyncCommitteeUpdates 1.7740 us/op 737.00 ns/op 2.41
mainnet_e81889 - altair afterProcessEpoch 230.64 ms/op 218.18 ms/op 1.06
phase0 processEpoch - mainnet_e58758 625.22 ms/op 634.36 ms/op 0.99
mainnet_e58758 - phase0 beforeProcessEpoch 262.54 ms/op 224.72 ms/op 1.17
mainnet_e58758 - phase0 processJustificationAndFinalization 34.194 us/op 15.852 us/op 2.16
mainnet_e58758 - phase0 processRewardsAndPenalties 148.98 ms/op 126.39 ms/op 1.18
mainnet_e58758 - phase0 processRegistryUpdates 19.322 us/op 7.7070 us/op 2.51
mainnet_e58758 - phase0 processSlashings 2.0150 us/op 620.00 ns/op 3.25
mainnet_e58758 - phase0 processEth1DataReset 2.3530 us/op 611.00 ns/op 3.85
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.5884 ms/op 1.8937 ms/op 1.37
mainnet_e58758 - phase0 processSlashingsReset 10.771 us/op 4.5280 us/op 2.38
mainnet_e58758 - phase0 processRandaoMixesReset 13.452 us/op 3.8470 us/op 3.50
mainnet_e58758 - phase0 processHistoricalRootsUpdate 2.2970 us/op 647.00 ns/op 3.55
mainnet_e58758 - phase0 processParticipationRecordUpdates 12.246 us/op 3.6570 us/op 3.35
mainnet_e58758 - phase0 afterProcessEpoch 187.85 ms/op 162.71 ms/op 1.15
phase0 processEffectiveBalanceUpdates - 250000 normalcase 3.1145 ms/op 2.1221 ms/op 1.47
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 3.6001 ms/op 2.2562 ms/op 1.60
altair processInactivityUpdates - 250000 normalcase 45.815 ms/op 51.798 ms/op 0.88
altair processInactivityUpdates - 250000 worstcase 59.941 ms/op 53.468 ms/op 1.12
phase0 processRegistryUpdates - 250000 normalcase 16.260 us/op 6.7660 us/op 2.40
phase0 processRegistryUpdates - 250000 badcase_full_deposits 540.18 us/op 374.51 us/op 1.44
phase0 processRegistryUpdates - 250000 worstcase 0.5 250.07 ms/op 227.53 ms/op 1.10
altair processRewardsAndPenalties - 250000 normalcase 155.16 ms/op 134.16 ms/op 1.16
altair processRewardsAndPenalties - 250000 worstcase 101.92 ms/op 135.14 ms/op 0.75
phase0 getAttestationDeltas - 250000 normalcase 16.377 ms/op 11.835 ms/op 1.38
phase0 getAttestationDeltas - 250000 worstcase 13.938 ms/op 11.712 ms/op 1.19
phase0 processSlashings - 250000 worstcase 6.2589 ms/op 5.0034 ms/op 1.25
altair processSyncCommitteeUpdates - 250000 339.18 ms/op 289.03 ms/op 1.17
BeaconState.hashTreeRoot - No change 554.00 ns/op 522.00 ns/op 1.06
BeaconState.hashTreeRoot - 1 full validator 70.272 us/op 73.151 us/op 0.96
BeaconState.hashTreeRoot - 32 full validator 650.18 us/op 718.76 us/op 0.90
BeaconState.hashTreeRoot - 512 full validator 7.4117 ms/op 8.7571 ms/op 0.85
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 95.785 us/op 101.28 us/op 0.95
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.6581 ms/op 1.3132 ms/op 1.26
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.212 ms/op 17.688 ms/op 0.97
BeaconState.hashTreeRoot - 1 balances 70.317 us/op 67.073 us/op 1.05
BeaconState.hashTreeRoot - 32 balances 695.40 us/op 735.50 us/op 0.95
BeaconState.hashTreeRoot - 512 balances 6.7723 ms/op 7.0486 ms/op 0.96
BeaconState.hashTreeRoot - 250000 balances 112.05 ms/op 118.12 ms/op 0.95
aggregationBits - 2048 els - zipIndexesInBitList 39.258 us/op 25.255 us/op 1.55
regular array get 100000 times 79.561 us/op 60.572 us/op 1.31
wrappedArray get 100000 times 81.017 us/op 60.609 us/op 1.34
arrayWithProxy get 100000 times 33.891 ms/op 28.945 ms/op 1.17
ssz.Root.equals 640.00 ns/op 475.00 ns/op 1.35
byteArrayEquals 627.00 ns/op 469.00 ns/op 1.34
shuffle list - 16384 els 13.117 ms/op 11.293 ms/op 1.16
shuffle list - 250000 els 193.32 ms/op 167.03 ms/op 1.16
processSlot - 1 slots 15.668 us/op 13.003 us/op 1.20
processSlot - 32 slots 2.2081 ms/op 1.9407 ms/op 1.14
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 1.4798 ms/op 382.59 us/op 3.87
getCommitteeAssignments - req 1 vs - 250000 vc 6.4451 ms/op 5.4642 ms/op 1.18
getCommitteeAssignments - req 100 vs - 250000 vc 8.8400 ms/op 7.9903 ms/op 1.11
getCommitteeAssignments - req 1000 vs - 250000 vc 9.4385 ms/op 8.5769 ms/op 1.10
computeProposers - vc 250000 19.717 ms/op 18.440 ms/op 1.07
computeEpochShuffling - vc 250000 197.24 ms/op 170.07 ms/op 1.16
getNextSyncCommittee - vc 250000 327.75 ms/op 281.33 ms/op 1.17

by benchmarkbot/action

@dapplion dapplion merged commit 938032c into unstable Jul 18, 2022
@dapplion dapplion deleted the dapplion/disable-workers-for-e2e-test branch July 18, 2022 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants