Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dont track getEffectiveBalanceIncrementsZeroInactive perf test #4383

Merged
merged 1 commit into from Aug 7, 2022
Merged

Dont track getEffectiveBalanceIncrementsZeroInactive perf test #4383

merged 1 commit into from Aug 7, 2022

Conversation

dapplion
Copy link
Contributor

@dapplion dapplion commented Aug 7, 2022 •
edited

Motivation

This performance test is very unstable in CI, triggering many false positives. See #4366 (comment)

Description

Dont track getEffectiveBalanceIncrementsZeroInactive perf test

@dapplion dapplion requested a review from a team as a code owner August 7, 2022 15:08
@github-actions
Copy link
Contributor

github-actions bot commented Aug 7, 2022

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 8d8c2ae Previous: ac5398e Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.2904 ms/op 2.1150 ms/op 1.08
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 75.188 us/op 72.465 us/op 1.04
BLS verify - blst-native 2.0033 ms/op 1.8562 ms/op 1.08
BLS verifyMultipleSignatures 3 - blst-native 3.9801 ms/op 3.8066 ms/op 1.05
BLS verifyMultipleSignatures 8 - blst-native 8.9508 ms/op 8.1726 ms/op 1.10
BLS verifyMultipleSignatures 32 - blst-native 31.289 ms/op 29.663 ms/op 1.05
BLS aggregatePubkeys 32 - blst-native 41.725 us/op 39.242 us/op 1.06
BLS aggregatePubkeys 128 - blst-native 162.59 us/op 152.28 us/op 1.07
getAttestationsForBlock 166.37 ms/op 168.61 ms/op 0.99
isKnown best case - 1 super set check 618.00 ns/op 453.00 ns/op 1.36
isKnown normal case - 2 super set checks 604.00 ns/op 442.00 ns/op 1.37
isKnown worse case - 16 super set checks 693.00 ns/op 441.00 ns/op 1.57
CheckpointStateCache - add get delete 10.519 us/op 8.6510 us/op 1.22
validate gossip signedAggregateAndProof - struct 5.1935 ms/op 4.2709 ms/op 1.22
validate gossip attestation - struct 2.4070 ms/op 2.0376 ms/op 1.18
altair verifyImport mainnet_s3766816:31 9.6248 s/op 8.6527 s/op 1.11
pickEth1Vote - no votes 2.6062 ms/op 2.1998 ms/op 1.18
pickEth1Vote - max votes 26.619 ms/op 20.760 ms/op 1.28
pickEth1Vote - Eth1Data hashTreeRoot value x2048 17.407 ms/op 11.068 ms/op 1.57
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 30.440 ms/op 22.937 ms/op 1.33
pickEth1Vote - Eth1Data fastSerialize value x2048 1.8439 ms/op 1.5514 ms/op 1.19
pickEth1Vote - Eth1Data fastSerialize tree x2048 16.905 ms/op 12.160 ms/op 1.39
bytes32 toHexString 1.2400 us/op 983.00 ns/op 1.26
bytes32 Buffer.toString(hex) 829.00 ns/op 680.00 ns/op 1.22
bytes32 Buffer.toString(hex) from Uint8Array 1.1250 us/op 891.00 ns/op 1.26
bytes32 Buffer.toString(hex) + 0x 868.00 ns/op 679.00 ns/op 1.28
Object access 1 prop 0.42300 ns/op 0.33600 ns/op 1.26
Map access 1 prop 0.36100 ns/op 0.29400 ns/op 1.23
Object get x1000 17.490 ns/op 17.938 ns/op 0.98
Map get x1000 1.1620 ns/op 0.99300 ns/op 1.17
Object set x1000 130.55 ns/op 111.27 ns/op 1.17
Map set x1000 88.646 ns/op 64.028 ns/op 1.38
Return object 10000 times 0.43990 ns/op 0.37830 ns/op 1.16
Throw Error 10000 times 7.0022 us/op 5.9375 us/op 1.18
enrSubnets - fastDeserialize 64 bits 3.0240 us/op 2.4580 us/op 1.23
enrSubnets - ssz BitVector 64 bits 846.00 ns/op 743.00 ns/op 1.14
enrSubnets - fastDeserialize 4 bits 439.00 ns/op 358.00 ns/op 1.23
enrSubnets - ssz BitVector 4 bits 841.00 ns/op 733.00 ns/op 1.15
prioritizePeers score -10:0 att 32-0.1 sync 2-0 115.25 us/op 88.211 us/op 1.31
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 148.44 us/op 124.83 us/op 1.19
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 258.45 us/op 201.31 us/op 1.28
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 554.23 us/op 426.50 us/op 1.30
prioritizePeers score 0:0 att 64-1 sync 4-1 518.79 us/op 464.83 us/op 1.12
RateTracker 1000000 limit, 1 obj count per request 166.06 ns/op 180.50 ns/op 0.92
RateTracker 1000000 limit, 2 obj count per request 131.57 ns/op 134.16 ns/op 0.98
RateTracker 1000000 limit, 4 obj count per request 101.64 ns/op 111.15 ns/op 0.91
RateTracker 1000000 limit, 8 obj count per request 92.960 ns/op 100.72 ns/op 0.92
RateTracker with prune 3.9670 us/op 3.8870 us/op 1.02
array of 16000 items push then shift 4.7360 us/op 3.1465 us/op 1.51
LinkedList of 16000 items push then shift 24.714 ns/op 25.446 ns/op 0.97
array of 16000 items push then pop 225.23 ns/op 242.93 ns/op 0.93
LinkedList of 16000 items push then pop 18.285 ns/op 21.849 ns/op 0.84
array of 24000 items push then shift 7.3309 us/op 4.5576 us/op 1.61
LinkedList of 24000 items push then shift 23.685 ns/op 28.964 ns/op 0.82
array of 24000 items push then pop 224.13 ns/op 212.72 ns/op 1.05
LinkedList of 24000 items push then pop 20.275 ns/op 21.822 ns/op 0.93
intersect bitArray bitLen 8 10.773 ns/op 11.765 ns/op 0.92
intersect array and set length 8 153.91 ns/op 155.18 ns/op 0.99
intersect bitArray bitLen 128 64.779 ns/op 61.971 ns/op 1.05
intersect array and set length 128 2.2738 us/op 2.1699 us/op 1.05
Buffer.concat 32 items 2.1680 ns/op 1.8610 ns/op 1.16
pass gossip attestations to forkchoice per slot 3.2580 ms/op 3.5572 ms/op 0.92
computeDeltas 4.2113 ms/op 4.0154 ms/op 1.05
computeProposerBoostScoreFromBalances 767.84 us/op 907.66 us/op 0.85
altair processAttestation - 250000 vs - 7PWei normalcase 4.7193 ms/op 3.9306 ms/op 1.20
altair processAttestation - 250000 vs - 7PWei worstcase 6.3448 ms/op 5.8488 ms/op 1.08
altair processAttestation - setStatus - 1/6 committees join 232.90 us/op 205.44 us/op 1.13
altair processAttestation - setStatus - 1/3 committees join 434.87 us/op 395.06 us/op 1.10
altair processAttestation - setStatus - 1/2 committees join 655.29 us/op 549.50 us/op 1.19
altair processAttestation - setStatus - 2/3 committees join 788.85 us/op 707.80 us/op 1.11
altair processAttestation - setStatus - 4/5 committees join 1.1810 ms/op 984.13 us/op 1.20
altair processAttestation - setStatus - 100% committees join 1.3386 ms/op 1.1654 ms/op 1.15
altair processBlock - 250000 vs - 7PWei normalcase 28.900 ms/op 28.588 ms/op 1.01
altair processBlock - 250000 vs - 7PWei normalcase hashState 40.979 ms/op 43.189 ms/op 0.95
altair processBlock - 250000 vs - 7PWei worstcase 98.738 ms/op 82.065 ms/op 1.20
altair processBlock - 250000 vs - 7PWei worstcase hashState 107.56 ms/op 104.92 ms/op 1.03
phase0 processBlock - 250000 vs - 7PWei normalcase 4.9707 ms/op 4.3230 ms/op 1.15
phase0 processBlock - 250000 vs - 7PWei worstcase 51.097 ms/op 46.980 ms/op 1.09
altair processEth1Data - 250000 vs - 7PWei normalcase 958.90 us/op 807.39 us/op 1.19
Tree 40 250000 create 798.95 ms/op 809.32 ms/op 0.99
Tree 40 250000 get(125000) 275.98 ns/op 285.14 ns/op 0.97
Tree 40 250000 set(125000) 2.6765 us/op 2.4585 us/op 1.09
Tree 40 250000 toArray() 31.690 ms/op 32.522 ms/op 0.97
Tree 40 250000 iterate all - toArray() + loop 30.752 ms/op 32.922 ms/op 0.93
Tree 40 250000 iterate all - get(i) 113.43 ms/op 111.36 ms/op 1.02
MutableVector 250000 create 14.362 ms/op 16.582 ms/op 0.87
MutableVector 250000 get(125000) 11.909 ns/op 14.752 ns/op 0.81
MutableVector 250000 set(125000) 764.64 ns/op 665.69 ns/op 1.15
MutableVector 250000 toArray() 6.8463 ms/op 7.7473 ms/op 0.88
MutableVector 250000 iterate all - toArray() + loop 7.3032 ms/op 7.7587 ms/op 0.94
MutableVector 250000 iterate all - get(i) 3.3151 ms/op 3.4389 ms/op 0.96
Array 250000 create 7.0483 ms/op 6.7398 ms/op 1.05
Array 250000 clone - spread 3.6375 ms/op 3.7597 ms/op 0.97
Array 250000 get(125000) 1.5640 ns/op 1.5150 ns/op 1.03
Array 250000 set(125000) 1.6330 ns/op 1.5380 ns/op 1.06
Array 250000 iterate all - loop 137.12 us/op 167.98 us/op 0.82
effectiveBalanceIncrements clone Uint8Array 300000 229.29 us/op 93.239 us/op 2.46
effectiveBalanceIncrements clone MutableVector 300000 731.00 ns/op 639.00 ns/op 1.14
effectiveBalanceIncrements rw all Uint8Array 300000 261.44 us/op 252.59 us/op 1.04
effectiveBalanceIncrements rw all MutableVector 300000 178.95 ms/op 163.41 ms/op 1.10
phase0 afterProcessEpoch - 250000 vs - 7PWei 185.30 ms/op 181.16 ms/op 1.02
phase0 beforeProcessEpoch - 250000 vs - 7PWei 76.692 ms/op 77.164 ms/op 0.99
altair processEpoch - mainnet_e81889 640.80 ms/op 590.85 ms/op 1.08
mainnet_e81889 - altair beforeProcessEpoch 103.01 ms/op 182.88 ms/op 0.56
mainnet_e81889 - altair processJustificationAndFinalization 70.753 us/op 32.992 us/op 2.14
mainnet_e81889 - altair processInactivityUpdates 9.5891 ms/op 11.508 ms/op 0.83
mainnet_e81889 - altair processRewardsAndPenalties 160.66 ms/op 94.500 ms/op 1.70
mainnet_e81889 - altair processRegistryUpdates 15.077 us/op 6.3390 us/op 2.38
mainnet_e81889 - altair processSlashings 4.4510 us/op 1.4510 us/op 3.07
mainnet_e81889 - altair processEth1DataReset 4.6400 us/op 1.5060 us/op 3.08
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.5482 ms/op 2.5229 ms/op 1.01
mainnet_e81889 - altair processSlashingsReset 30.169 us/op 8.5740 us/op 3.52
mainnet_e81889 - altair processRandaoMixesReset 28.078 us/op 9.0430 us/op 3.10
mainnet_e81889 - altair processHistoricalRootsUpdate 4.4860 us/op 1.6370 us/op 2.74
mainnet_e81889 - altair processParticipationFlagUpdates 17.697 us/op 6.0790 us/op 2.91
mainnet_e81889 - altair processSyncCommitteeUpdates 3.6280 us/op 1.2290 us/op 2.95
mainnet_e81889 - altair afterProcessEpoch 180.20 ms/op 192.81 ms/op 0.93
phase0 processEpoch - mainnet_e58758 541.26 ms/op 540.56 ms/op 1.00
mainnet_e58758 - phase0 beforeProcessEpoch 284.42 ms/op 234.26 ms/op 1.21
mainnet_e58758 - phase0 processJustificationAndFinalization 71.554 us/op 34.345 us/op 2.08
mainnet_e58758 - phase0 processRewardsAndPenalties 114.58 ms/op 142.03 ms/op 0.81
mainnet_e58758 - phase0 processRegistryUpdates 38.677 us/op 16.841 us/op 2.30
mainnet_e58758 - phase0 processSlashings 3.1570 us/op 1.4440 us/op 2.19
mainnet_e58758 - phase0 processEth1DataReset 3.8170 us/op 1.2860 us/op 2.97
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.1269 ms/op 2.0305 ms/op 1.05
mainnet_e58758 - phase0 processSlashingsReset 18.543 us/op 7.2210 us/op 2.57
mainnet_e58758 - phase0 processRandaoMixesReset 27.995 us/op 9.7740 us/op 2.86
mainnet_e58758 - phase0 processHistoricalRootsUpdate 4.6590 us/op 1.6240 us/op 2.87
mainnet_e58758 - phase0 processParticipationRecordUpdates 23.314 us/op 9.5860 us/op 2.43
mainnet_e58758 - phase0 afterProcessEpoch 149.84 ms/op 157.92 ms/op 0.95
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.3352 ms/op 2.7741 ms/op 0.84
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.4301 ms/op 3.3698 ms/op 0.72
altair processInactivityUpdates - 250000 normalcase 48.769 ms/op 41.999 ms/op 1.16
altair processInactivityUpdates - 250000 worstcase 50.358 ms/op 51.024 ms/op 0.99
phase0 processRegistryUpdates - 250000 normalcase 29.098 us/op 14.277 us/op 2.04
phase0 processRegistryUpdates - 250000 badcase_full_deposits 456.64 us/op 409.00 us/op 1.12
phase0 processRegistryUpdates - 250000 worstcase 0.5 232.17 ms/op 220.90 ms/op 1.05
altair processRewardsAndPenalties - 250000 normalcase 95.298 ms/op 138.69 ms/op 0.69
altair processRewardsAndPenalties - 250000 worstcase 124.33 ms/op 115.03 ms/op 1.08
phase0 getAttestationDeltas - 250000 normalcase 14.227 ms/op 13.400 ms/op 1.06
phase0 getAttestationDeltas - 250000 worstcase 14.164 ms/op 13.369 ms/op 1.06
phase0 processSlashings - 250000 worstcase 6.7466 ms/op 5.3654 ms/op 1.26
altair processSyncCommitteeUpdates - 250000 335.65 ms/op 284.00 ms/op 1.18
BeaconState.hashTreeRoot - No change 653.00 ns/op 499.00 ns/op 1.31
BeaconState.hashTreeRoot - 1 full validator 76.922 us/op 63.011 us/op 1.22
BeaconState.hashTreeRoot - 32 full validator 775.78 us/op 623.17 us/op 1.24
BeaconState.hashTreeRoot - 512 full validator 7.4368 ms/op 6.4091 ms/op 1.16
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 100.70 us/op 82.583 us/op 1.22
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3964 ms/op 1.1257 ms/op 1.24
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.091 ms/op 15.030 ms/op 1.14
BeaconState.hashTreeRoot - 1 balances 64.161 us/op 56.355 us/op 1.14
BeaconState.hashTreeRoot - 32 balances 603.97 us/op 574.02 us/op 1.05
BeaconState.hashTreeRoot - 512 balances 6.4129 ms/op 5.7063 ms/op 1.12
BeaconState.hashTreeRoot - 250000 balances 92.930 ms/op 91.078 ms/op 1.02
aggregationBits - 2048 els - zipIndexesInBitList 29.426 us/op 36.583 us/op 0.80
regular array get 100000 times 50.059 us/op 67.412 us/op 0.74
wrappedArray get 100000 times 49.281 us/op 67.430 us/op 0.73
arrayWithProxy get 100000 times 30.766 ms/op 28.903 ms/op 1.06
ssz.Root.equals 467.00 ns/op 488.00 ns/op 0.96
byteArrayEquals 469.00 ns/op 484.00 ns/op 0.97
shuffle list - 16384 els 11.052 ms/op 12.291 ms/op 0.90
shuffle list - 250000 els 157.09 ms/op 161.51 ms/op 0.97
processSlot - 1 slots 15.664 us/op 11.768 us/op 1.33
processSlot - 32 slots 2.0824 ms/op 1.6843 ms/op 1.24
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 508.94 us/op 399.71 us/op 1.27
getCommitteeAssignments - req 1 vs - 250000 vc 5.0251 ms/op 5.2711 ms/op 0.95
getCommitteeAssignments - req 100 vs - 250000 vc 7.4512 ms/op 7.3323 ms/op 1.02
getCommitteeAssignments - req 1000 vs - 250000 vc 7.4691 ms/op 7.8433 ms/op 0.95
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 8.9400 ns/op
state getBlockRootAtSlot - 250000 vs - 7PWei 1.1932 us/op
computeProposers - vc 250000 19.329 ms/op 18.178 ms/op 1.06
computeEpochShuffling - vc 250000 168.26 ms/op 165.40 ms/op 1.02
getNextSyncCommittee - vc 250000 337.35 ms/op 267.84 ms/op 1.26

by benchmarkbot/action

@wemeetagain wemeetagain merged commit b9c6bb1 into unstable Aug 7, 2022
@wemeetagain wemeetagain deleted the dapplion/bench-getEffectiveBalanceIncrementsZeroInactive branch August 7, 2022 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@dapplion @wemeetagain