Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix unstable sim tests #4616

Merged
merged 14 commits into from Oct 4, 2022
Merged

Fix unstable sim tests #4616

merged 14 commits into from Oct 4, 2022

Conversation

nazarhussain
Copy link
Contributor

Motivation

  • Make the assertions more flexible
  • Wait for block from node not the absolute clock time.

Closes #4609

Steps to test or reproduce

Run all CI jobs all together.

@nazarhussain nazarhussain requested a review from a team as a code owner September 30, 2022 08:17
@nazarhussain nazarhussain self-assigned this Sep 30, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Sep 30, 2022 •
edited

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 6505d18 Previous: 19b5fe1 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.3007 ms/op 3.1945 ms/op 0.72
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 67.085 us/op 120.58 us/op 0.56
BLS verify - blst-native 2.1641 ms/op 1.8641 ms/op 1.16
BLS verifyMultipleSignatures 3 - blst-native 4.4726 ms/op 3.8090 ms/op 1.17
BLS verifyMultipleSignatures 8 - blst-native 9.6654 ms/op 8.2041 ms/op 1.18
BLS verifyMultipleSignatures 32 - blst-native 35.139 ms/op 29.739 ms/op 1.18
BLS aggregatePubkeys 32 - blst-native 46.506 us/op 40.198 us/op 1.16
BLS aggregatePubkeys 128 - blst-native 181.74 us/op 153.59 us/op 1.18
getAttestationsForBlock 82.256 ms/op 113.62 ms/op 0.72
isKnown best case - 1 super set check 480.00 ns/op 427.00 ns/op 1.12
isKnown normal case - 2 super set checks 470.00 ns/op 417.00 ns/op 1.13
isKnown worse case - 16 super set checks 472.00 ns/op 414.00 ns/op 1.14
CheckpointStateCache - add get delete 9.3330 us/op 10.281 us/op 0.91
validate gossip signedAggregateAndProof - struct 5.0177 ms/op 4.2667 ms/op 1.18
validate gossip attestation - struct 2.3819 ms/op 2.0377 ms/op 1.17
pickEth1Vote - no votes 2.1865 ms/op 2.3363 ms/op 0.94
pickEth1Vote - max votes 21.217 ms/op 25.415 ms/op 0.83
pickEth1Vote - Eth1Data hashTreeRoot value x2048 13.180 ms/op 11.976 ms/op 1.10
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 22.482 ms/op 23.227 ms/op 0.97
pickEth1Vote - Eth1Data fastSerialize value x2048 1.4967 ms/op 1.5993 ms/op 0.94
pickEth1Vote - Eth1Data fastSerialize tree x2048 15.198 ms/op 16.162 ms/op 0.94
bytes32 toHexString 1.0580 us/op 1.1270 us/op 0.94
bytes32 Buffer.toString(hex) 823.00 ns/op 709.00 ns/op 1.16
bytes32 Buffer.toString(hex) from Uint8Array 1.0600 us/op 973.00 ns/op 1.09
bytes32 Buffer.toString(hex) + 0x 830.00 ns/op 703.00 ns/op 1.18
Object access 1 prop 0.39000 ns/op 0.36700 ns/op 1.06
Map access 1 prop 0.30400 ns/op 0.29900 ns/op 1.02
Object get x1000 10.798 ns/op 18.390 ns/op 0.59
Map get x1000 0.94700 ns/op 1.0040 ns/op 0.94
Object set x1000 80.912 ns/op 142.14 ns/op 0.57
Map set x1000 53.214 ns/op 81.092 ns/op 0.66
Return object 10000 times 0.43330 ns/op 0.37870 ns/op 1.14
Throw Error 10000 times 6.0371 us/op 6.0074 us/op 1.00
enrSubnets - fastDeserialize 64 bits 2.7690 us/op 2.9560 us/op 0.94
enrSubnets - ssz BitVector 64 bits 810.00 ns/op 819.00 ns/op 0.99
enrSubnets - fastDeserialize 4 bits 419.00 ns/op 379.00 ns/op 1.11
enrSubnets - ssz BitVector 4 bits 803.00 ns/op 819.00 ns/op 0.98
prioritizePeers score -10:0 att 32-0.1 sync 2-0 88.003 us/op 97.761 us/op 0.90
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 126.34 us/op 127.79 us/op 0.99
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 220.53 us/op 214.48 us/op 1.03
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 337.67 us/op 516.50 us/op 0.65
prioritizePeers score 0:0 att 64-1 sync 4-1 407.42 us/op 479.41 us/op 0.85
RateTracker 1000000 limit, 1 obj count per request 188.27 ns/op 210.24 ns/op 0.90
RateTracker 1000000 limit, 2 obj count per request 136.72 ns/op 164.99 ns/op 0.83
RateTracker 1000000 limit, 4 obj count per request 114.32 ns/op 140.98 ns/op 0.81
RateTracker 1000000 limit, 8 obj count per request 103.48 ns/op 129.13 ns/op 0.80
RateTracker with prune 4.4760 us/op 6.6680 us/op 0.67
array of 16000 items push then shift 51.589 us/op 3.2785 us/op 15.74
LinkedList of 16000 items push then shift 12.645 ns/op 21.273 ns/op 0.59
array of 16000 items push then pop 209.63 ns/op 319.12 ns/op 0.66
LinkedList of 16000 items push then pop 12.183 ns/op 18.565 ns/op 0.66
array of 24000 items push then shift 77.368 us/op 4.6309 us/op 16.71
LinkedList of 24000 items push then shift 13.308 ns/op 30.777 ns/op 0.43
array of 24000 items push then pop 197.14 ns/op 274.80 ns/op 0.72
LinkedList of 24000 items push then pop 12.276 ns/op 21.964 ns/op 0.56
intersect bitArray bitLen 8 10.780 ns/op 11.606 ns/op 0.93
intersect array and set length 8 139.93 ns/op 212.39 ns/op 0.66
intersect bitArray bitLen 128 57.778 ns/op 72.217 ns/op 0.80
intersect array and set length 128 1.8409 us/op 2.5535 us/op 0.72
Buffer.concat 32 items 2.1100 ns/op 2.0570 ns/op 1.03
pass gossip attestations to forkchoice per slot 6.3795 ms/op 4.2871 ms/op 1.49
computeDeltas 4.9531 ms/op 5.9766 ms/op 0.83
computeProposerBoostScoreFromBalances 807.71 us/op 908.13 us/op 0.89
altair processAttestation - 250000 vs - 7PWei normalcase 3.9008 ms/op 4.2577 ms/op 0.92
altair processAttestation - 250000 vs - 7PWei worstcase 5.5717 ms/op 6.8593 ms/op 0.81
altair processAttestation - setStatus - 1/6 committees join 190.01 us/op 238.41 us/op 0.80
altair processAttestation - setStatus - 1/3 committees join 357.22 us/op 448.58 us/op 0.80
altair processAttestation - setStatus - 1/2 committees join 510.39 us/op 631.21 us/op 0.81
altair processAttestation - setStatus - 2/3 committees join 667.80 us/op 796.35 us/op 0.84
altair processAttestation - setStatus - 4/5 committees join 938.56 us/op 1.1369 ms/op 0.83
altair processAttestation - setStatus - 100% committees join 1.1265 ms/op 1.3308 ms/op 0.85
altair processBlock - 250000 vs - 7PWei normalcase 24.400 ms/op 37.865 ms/op 0.64
altair processBlock - 250000 vs - 7PWei normalcase hashState 38.993 ms/op 51.034 ms/op 0.76
altair processBlock - 250000 vs - 7PWei worstcase 82.020 ms/op 92.713 ms/op 0.88
altair processBlock - 250000 vs - 7PWei worstcase hashState 105.06 ms/op 112.74 ms/op 0.93
phase0 processBlock - 250000 vs - 7PWei normalcase 7.2075 ms/op 4.0322 ms/op 1.79
phase0 processBlock - 250000 vs - 7PWei worstcase 51.763 ms/op 47.663 ms/op 1.09
altair processEth1Data - 250000 vs - 7PWei normalcase 804.78 us/op 1.0064 ms/op 0.80
Tree 40 250000 create 772.37 ms/op 1.1071 s/op 0.70
Tree 40 250000 get(125000) 236.82 ns/op 301.55 ns/op 0.79
Tree 40 250000 set(125000) 2.4221 us/op 2.9014 us/op 0.83
Tree 40 250000 toArray() 28.253 ms/op 41.757 ms/op 0.68
Tree 40 250000 iterate all - toArray() + loop 28.390 ms/op 41.541 ms/op 0.68
Tree 40 250000 iterate all - get(i) 113.26 ms/op 122.91 ms/op 0.92
MutableVector 250000 create 14.382 ms/op 16.493 ms/op 0.87
MutableVector 250000 get(125000) 10.639 ns/op 14.758 ns/op 0.72
MutableVector 250000 set(125000) 610.24 ns/op 855.48 ns/op 0.71
MutableVector 250000 toArray() 6.9828 ms/op 10.320 ms/op 0.68
MutableVector 250000 iterate all - toArray() + loop 7.1548 ms/op 10.285 ms/op 0.70
MutableVector 250000 iterate all - get(i) 2.8063 ms/op 3.2859 ms/op 0.85
Array 250000 create 6.0098 ms/op 9.4157 ms/op 0.64
Array 250000 clone - spread 2.4751 ms/op 4.9419 ms/op 0.50
Array 250000 get(125000) 1.1730 ns/op 1.8570 ns/op 0.63
Array 250000 set(125000) 1.1620 ns/op 1.7050 ns/op 0.68
Array 250000 iterate all - loop 151.93 us/op 167.97 us/op 0.90
effectiveBalanceIncrements clone Uint8Array 300000 43.124 us/op 162.22 us/op 0.27
effectiveBalanceIncrements clone MutableVector 300000 720.00 ns/op 1.2090 us/op 0.60
effectiveBalanceIncrements rw all Uint8Array 300000 247.49 us/op 252.76 us/op 0.98
effectiveBalanceIncrements rw all MutableVector 300000 142.13 ms/op 292.57 ms/op 0.49
phase0 afterProcessEpoch - 250000 vs - 7PWei 190.06 ms/op 191.35 ms/op 0.99
phase0 beforeProcessEpoch - 250000 vs - 7PWei 63.346 ms/op 114.32 ms/op 0.55
altair processEpoch - mainnet_e81889 565.20 ms/op 674.53 ms/op 0.84
mainnet_e81889 - altair beforeProcessEpoch 128.31 ms/op 177.17 ms/op 0.72
mainnet_e81889 - altair processJustificationAndFinalization 19.529 us/op 22.658 us/op 0.86
mainnet_e81889 - altair processInactivityUpdates 8.6803 ms/op 12.757 ms/op 0.68
mainnet_e81889 - altair processRewardsAndPenalties 80.238 ms/op 111.53 ms/op 0.72
mainnet_e81889 - altair processRegistryUpdates 2.5030 us/op 3.6760 us/op 0.68
mainnet_e81889 - altair processSlashings 517.00 ns/op 817.00 ns/op 0.63
mainnet_e81889 - altair processEth1DataReset 565.00 ns/op 904.00 ns/op 0.63
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.9384 ms/op 2.4198 ms/op 0.80
mainnet_e81889 - altair processSlashingsReset 4.5760 us/op 5.6050 us/op 0.82
mainnet_e81889 - altair processRandaoMixesReset 4.3310 us/op 8.7880 us/op 0.49
mainnet_e81889 - altair processHistoricalRootsUpdate 580.00 ns/op 1.5990 us/op 0.36
mainnet_e81889 - altair processParticipationFlagUpdates 12.672 us/op 3.9660 us/op 3.20
mainnet_e81889 - altair processSyncCommitteeUpdates 677.00 ns/op 961.00 ns/op 0.70
mainnet_e81889 - altair afterProcessEpoch 220.99 ms/op 210.52 ms/op 1.05
phase0 processEpoch - mainnet_e58758 592.20 ms/op 733.22 ms/op 0.81
mainnet_e58758 - phase0 beforeProcessEpoch 157.94 ms/op 279.23 ms/op 0.57
mainnet_e58758 - phase0 processJustificationAndFinalization 17.015 us/op 22.034 us/op 0.77
mainnet_e58758 - phase0 processRewardsAndPenalties 103.03 ms/op 124.44 ms/op 0.83
mainnet_e58758 - phase0 processRegistryUpdates 7.5740 us/op 10.255 us/op 0.74
mainnet_e58758 - phase0 processSlashings 592.00 ns/op 745.00 ns/op 0.79
mainnet_e58758 - phase0 processEth1DataReset 645.00 ns/op 781.00 ns/op 0.83
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.6538 ms/op 2.0008 ms/op 0.83
mainnet_e58758 - phase0 processSlashingsReset 3.3420 us/op 4.6480 us/op 0.72
mainnet_e58758 - phase0 processRandaoMixesReset 4.2650 us/op 5.6890 us/op 0.75
mainnet_e58758 - phase0 processHistoricalRootsUpdate 667.00 ns/op 935.00 ns/op 0.71
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.9990 us/op 5.1740 us/op 0.77
mainnet_e58758 - phase0 afterProcessEpoch 164.61 ms/op 168.51 ms/op 0.98
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.0284 ms/op 2.7531 ms/op 0.74
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.2585 ms/op 3.6656 ms/op 0.62
altair processInactivityUpdates - 250000 normalcase 41.349 ms/op 48.790 ms/op 0.85
altair processInactivityUpdates - 250000 worstcase 33.628 ms/op 60.180 ms/op 0.56
phase0 processRegistryUpdates - 250000 normalcase 6.6320 us/op 8.4930 us/op 0.78
phase0 processRegistryUpdates - 250000 badcase_full_deposits 379.56 us/op 415.65 us/op 0.91
phase0 processRegistryUpdates - 250000 worstcase 0.5 179.97 ms/op 269.35 ms/op 0.67
altair processRewardsAndPenalties - 250000 normalcase 77.907 ms/op 150.33 ms/op 0.52
altair processRewardsAndPenalties - 250000 worstcase 114.02 ms/op 104.01 ms/op 1.10
phase0 getAttestationDeltas - 250000 normalcase 12.499 ms/op 15.462 ms/op 0.81
phase0 getAttestationDeltas - 250000 worstcase 12.462 ms/op 14.478 ms/op 0.86
phase0 processSlashings - 250000 worstcase 5.0965 ms/op 5.2551 ms/op 0.97
altair processSyncCommitteeUpdates - 250000 301.44 ms/op 305.24 ms/op 0.99
BeaconState.hashTreeRoot - No change 504.00 ns/op 514.00 ns/op 0.98
BeaconState.hashTreeRoot - 1 full validator 71.638 us/op 67.731 us/op 1.06
BeaconState.hashTreeRoot - 32 full validator 710.20 us/op 667.62 us/op 1.06
BeaconState.hashTreeRoot - 512 full validator 7.0689 ms/op 6.9923 ms/op 1.01
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 87.186 us/op 85.382 us/op 1.02
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3011 ms/op 1.2647 ms/op 1.03
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.539 ms/op 17.108 ms/op 1.03
BeaconState.hashTreeRoot - 1 balances 70.612 us/op 64.899 us/op 1.09
BeaconState.hashTreeRoot - 32 balances 648.12 us/op 643.34 us/op 1.01
BeaconState.hashTreeRoot - 512 balances 6.9954 ms/op 6.7537 ms/op 1.04
BeaconState.hashTreeRoot - 250000 balances 100.90 ms/op 100.98 ms/op 1.00
aggregationBits - 2048 els - zipIndexesInBitList 27.587 us/op 43.239 us/op 0.64
regular array get 100000 times 61.000 us/op 67.481 us/op 0.90
wrappedArray get 100000 times 60.860 us/op 67.560 us/op 0.90
arrayWithProxy get 100000 times 29.021 ms/op 29.743 ms/op 0.98
ssz.Root.equals 490.00 ns/op 559.00 ns/op 0.88
byteArrayEquals 495.00 ns/op 539.00 ns/op 0.92
shuffle list - 16384 els 11.496 ms/op 11.085 ms/op 1.04
shuffle list - 250000 els 168.35 ms/op 167.50 ms/op 1.01
processSlot - 1 slots 13.787 us/op 14.047 us/op 0.98
processSlot - 32 slots 2.0280 ms/op 2.0176 ms/op 1.01
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 357.42 us/op 442.19 us/op 0.81
getCommitteeAssignments - req 1 vs - 250000 vc 5.4984 ms/op 5.3280 ms/op 1.03
getCommitteeAssignments - req 100 vs - 250000 vc 8.0431 ms/op 7.3347 ms/op 1.10
getCommitteeAssignments - req 1000 vs - 250000 vc 8.6130 ms/op 7.7905 ms/op 1.11
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 9.0500 ns/op 11.000 ns/op 0.82
state getBlockRootAtSlot - 250000 vs - 7PWei 1.1862 us/op 1.1644 us/op 1.02
computeProposers - vc 250000 18.374 ms/op 17.965 ms/op 1.02
computeEpochShuffling - vc 250000 172.56 ms/op 172.07 ms/op 1.00
getNextSyncCommittee - vc 250000 306.55 ms/op 291.21 ms/op 1.05

by benchmarkbot/action

@wemeetagain
Copy link
Member

@nazarhussain sim tests still failing

@dapplion
Copy link
Contributor

dapplion commented Oct 2, 2022

Sim tests still failing @nazarhussain

@nazarhussain
Copy link
Contributor Author

@wemeetagain @dapplion Yes I forget to mark this PR draft. It's not ready yet, created PR to run CI jobs.

@nazarhussain nazarhussain marked this pull request as draft October 4, 2022 08:06
dapplion
dapplion previously approved these changes Oct 4, 2022
View reviewed changes
@dapplion dapplion marked this pull request as ready for review October 4, 2022 12:21
@dapplion dapplion changed the title Fix unsable sim tests Fix unstable sim tests Oct 4, 2022
@dapplion dapplion enabled auto-merge (squash) October 4, 2022 13:10
@dapplion
Copy link
Contributor

dapplion commented Oct 4, 2022

Note: This branch also suffers from the socket timeout then disconnect issue at random, as #4621

Eph -1/6 1.002[NODE-2]           �[32minfo�[39m: Synced - slot: -2 - head: 0 0xe228…e2a5 - finalized: 0x0000…0000:0 - peers: 3
Eph -1/7 0.717[NODE-2/network] �[36mverbose�[39m: peer disconnected peer=16...SAdSwE, direction=outbound, status=CLOSED
Eph -1/7 0.961[NODE-2/network] �[36mverbose�[39m: peer disconnected peer=16...URw6iT, direction=outbound, status=CLOSED
Eph -1/7 1.001[NODE-2]           �[33mwarn�[39m: Low peer count peers=1
Eph -1/7 1.001[NODE-2/chain]  �[36mverbose�[39m: Clock slot slot=-1

@dapplion dapplion merged commit edb327d into unstable Oct 4, 2022
@dapplion dapplion deleted the nh/4609-sim-tests branch October 4, 2022 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

Assignees

@nazarhussain nazarhussain

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

SIM tests are unstable and causing delay in CI
3 participants
@nazarhussain @wemeetagain @dapplion