Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: update builder mergemock sim with builder block assertions #5432

Merged
merged 3 commits into from May 4, 2023
Merged

fix: update builder mergemock sim with builder block assertions #5432

merged 3 commits into from May 4, 2023

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Apr 28, 2023

Previously builder sim was running without any validations.

Modified the mergemock builder interaction to enable differentiating by fee recipient and use that to add engine and builder block assertions

@g11tech g11tech requested a review from a team as a code owner April 28, 2023 10:25
@g11tech g11tech changed the title fix: Update builder mergemock sim with builder block assertions fix: update builder mergemock sim with builder block assertions Apr 28, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Apr 28, 2023 •
edited

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: e8e76c1 Previous: d835c8c Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 863.50 us/op 571.40 us/op 1.51
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 57.963 us/op 45.329 us/op 1.28
BLS verify - blst-native 1.2503 ms/op 1.2293 ms/op 1.02
BLS verifyMultipleSignatures 3 - blst-native 2.5846 ms/op 2.4818 ms/op 1.04
BLS verifyMultipleSignatures 8 - blst-native 5.4839 ms/op 5.3084 ms/op 1.03
BLS verifyMultipleSignatures 32 - blst-native 19.760 ms/op 18.979 ms/op 1.04
BLS aggregatePubkeys 32 - blst-native 26.365 us/op 25.925 us/op 1.02
BLS aggregatePubkeys 128 - blst-native 105.45 us/op 101.56 us/op 1.04
getAttestationsForBlock 62.941 ms/op 56.021 ms/op 1.12
isKnown best case - 1 super set check 281.00 ns/op 256.00 ns/op 1.10
isKnown normal case - 2 super set checks 259.00 ns/op 255.00 ns/op 1.02
isKnown worse case - 16 super set checks 265.00 ns/op 252.00 ns/op 1.05
CheckpointStateCache - add get delete 5.8790 us/op 5.1170 us/op 1.15
validate gossip signedAggregateAndProof - struct 2.8513 ms/op 2.8275 ms/op 1.01
validate gossip attestation - struct 1.3619 ms/op 1.3541 ms/op 1.01
pickEth1Vote - no votes 1.4040 ms/op 1.2605 ms/op 1.11
pickEth1Vote - max votes 11.559 ms/op 11.477 ms/op 1.01
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.9279 ms/op 9.2133 ms/op 1.08
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 17.302 ms/op 19.569 ms/op 0.88
pickEth1Vote - Eth1Data fastSerialize value x2048 925.95 us/op 676.64 us/op 1.37
pickEth1Vote - Eth1Data fastSerialize tree x2048 8.7708 ms/op 8.9303 ms/op 0.98
bytes32 toHexString 853.00 ns/op 497.00 ns/op 1.72
bytes32 Buffer.toString(hex) 493.00 ns/op 364.00 ns/op 1.35
bytes32 Buffer.toString(hex) from Uint8Array 710.00 ns/op 568.00 ns/op 1.25
bytes32 Buffer.toString(hex) + 0x 459.00 ns/op 348.00 ns/op 1.32
Object access 1 prop 0.20300 ns/op 0.16200 ns/op 1.25
Map access 1 prop 0.16400 ns/op 0.15300 ns/op 1.07
Object get x1000 7.0360 ns/op 6.6350 ns/op 1.06
Map get x1000 0.64000 ns/op 0.61200 ns/op 1.05
Object set x1000 75.873 ns/op 53.324 ns/op 1.42
Map set x1000 57.690 ns/op 45.492 ns/op 1.27
Return object 10000 times 0.24980 ns/op 0.23600 ns/op 1.06
Throw Error 10000 times 4.4250 us/op 4.2968 us/op 1.03
fastMsgIdFn sha256 / 200 bytes 3.8340 us/op 3.5720 us/op 1.07
fastMsgIdFn h32 xxhash / 200 bytes 320.00 ns/op 299.00 ns/op 1.07
fastMsgIdFn h64 xxhash / 200 bytes 425.00 ns/op 444.00 ns/op 0.96
fastMsgIdFn sha256 / 1000 bytes 12.110 us/op 12.408 us/op 0.98
fastMsgIdFn h32 xxhash / 1000 bytes 433.00 ns/op 410.00 ns/op 1.06
fastMsgIdFn h64 xxhash / 1000 bytes 534.00 ns/op 467.00 ns/op 1.14
fastMsgIdFn sha256 / 10000 bytes 108.15 us/op 105.39 us/op 1.03
fastMsgIdFn h32 xxhash / 10000 bytes 2.0390 us/op 1.9170 us/op 1.06
fastMsgIdFn h64 xxhash / 10000 bytes 1.4630 us/op 1.3560 us/op 1.08
enrSubnets - fastDeserialize 64 bits 1.9300 us/op 1.3130 us/op 1.47
enrSubnets - ssz BitVector 64 bits 688.00 ns/op 502.00 ns/op 1.37
enrSubnets - fastDeserialize 4 bits 219.00 ns/op 168.00 ns/op 1.30
enrSubnets - ssz BitVector 4 bits 746.00 ns/op 515.00 ns/op 1.45
prioritizePeers score -10:0 att 32-0.1 sync 2-0 126.54 us/op 124.75 us/op 1.01
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 177.52 us/op 165.26 us/op 1.07
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 227.01 us/op 161.87 us/op 1.40
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 417.52 us/op 309.69 us/op 1.35
prioritizePeers score 0:0 att 64-1 sync 4-1 484.86 us/op 380.58 us/op 1.27
array of 16000 items push then shift 1.7418 us/op 1.6951 us/op 1.03
LinkedList of 16000 items push then shift 10.582 ns/op 9.1390 ns/op 1.16
array of 16000 items push then pop 136.05 ns/op 89.125 ns/op 1.53
LinkedList of 16000 items push then pop 10.805 ns/op 8.8940 ns/op 1.21
array of 24000 items push then shift 2.7647 us/op 2.4692 us/op 1.12
LinkedList of 24000 items push then shift 15.240 ns/op 10.688 ns/op 1.43
array of 24000 items push then pop 106.91 ns/op 77.337 ns/op 1.38
LinkedList of 24000 items push then pop 9.9190 ns/op 8.9520 ns/op 1.11
intersect bitArray bitLen 8 14.063 ns/op 13.640 ns/op 1.03
intersect array and set length 8 117.73 ns/op 76.267 ns/op 1.54
intersect bitArray bitLen 128 51.692 ns/op 44.457 ns/op 1.16
intersect array and set length 128 1.4980 us/op 1.0625 us/op 1.41
Buffer.concat 32 items 3.0530 us/op 2.6940 us/op 1.13
Uint8Array.set 32 items 3.5230 us/op 3.0510 us/op 1.15
pass gossip attestations to forkchoice per slot 4.2488 ms/op 3.4857 ms/op 1.22
computeDeltas 3.9132 ms/op 3.6943 ms/op 1.06
computeProposerBoostScoreFromBalances 1.9174 ms/op 1.8645 ms/op 1.03
altair processAttestation - 250000 vs - 7PWei normalcase 3.2180 ms/op 2.5024 ms/op 1.29
altair processAttestation - 250000 vs - 7PWei worstcase 4.2376 ms/op 3.8765 ms/op 1.09
altair processAttestation - setStatus - 1/6 committees join 147.57 us/op 153.97 us/op 0.96
altair processAttestation - setStatus - 1/3 committees join 291.50 us/op 305.16 us/op 0.96
altair processAttestation - setStatus - 1/2 committees join 389.02 us/op 392.91 us/op 0.99
altair processAttestation - setStatus - 2/3 committees join 508.41 us/op 487.18 us/op 1.04
altair processAttestation - setStatus - 4/5 committees join 689.26 us/op 690.36 us/op 1.00
altair processAttestation - setStatus - 100% committees join 805.87 us/op 793.61 us/op 1.02
altair processBlock - 250000 vs - 7PWei normalcase 21.283 ms/op 20.935 ms/op 1.02
altair processBlock - 250000 vs - 7PWei normalcase hashState 27.856 ms/op 31.357 ms/op 0.89
altair processBlock - 250000 vs - 7PWei worstcase 49.843 ms/op 61.625 ms/op 0.81
altair processBlock - 250000 vs - 7PWei worstcase hashState 82.986 ms/op 78.622 ms/op 1.06
phase0 processBlock - 250000 vs - 7PWei normalcase 2.5725 ms/op 2.1915 ms/op 1.17
phase0 processBlock - 250000 vs - 7PWei worstcase 33.386 ms/op 31.168 ms/op 1.07
altair processEth1Data - 250000 vs - 7PWei normalcase 666.65 us/op 490.93 us/op 1.36
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 7.7530 us/op 9.5350 us/op 0.81
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 26.263 us/op 28.506 us/op 0.92
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 10.929 us/op 9.9190 us/op 1.10
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 8.2940 us/op 9.3970 us/op 0.88
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 105.85 us/op 114.34 us/op 0.93
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 662.99 us/op 845.27 us/op 0.78
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 924.03 us/op 1.0843 ms/op 0.85
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 870.31 us/op 1.1244 ms/op 0.77
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.3434 ms/op 2.8698 ms/op 0.82
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.6787 ms/op 1.9502 ms/op 0.86
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 3.9148 ms/op 4.8816 ms/op 0.80
Tree 40 250000 create 305.46 ms/op 375.94 ms/op 0.81
Tree 40 250000 get(125000) 185.13 ns/op 203.73 ns/op 0.91
Tree 40 250000 set(125000) 914.80 ns/op 1.3145 us/op 0.70
Tree 40 250000 toArray() 18.071 ms/op 24.612 ms/op 0.73
Tree 40 250000 iterate all - toArray() + loop 17.970 ms/op 22.857 ms/op 0.79
Tree 40 250000 iterate all - get(i) 66.847 ms/op 81.869 ms/op 0.82
MutableVector 250000 create 10.748 ms/op 15.338 ms/op 0.70
MutableVector 250000 get(125000) 6.3210 ns/op 7.2690 ns/op 0.87
MutableVector 250000 set(125000) 255.42 ns/op 297.82 ns/op 0.86
MutableVector 250000 toArray() 2.6658 ms/op 3.9289 ms/op 0.68
MutableVector 250000 iterate all - toArray() + loop 2.7941 ms/op 3.5075 ms/op 0.80
MutableVector 250000 iterate all - get(i) 1.4783 ms/op 1.6052 ms/op 0.92
Array 250000 create 2.5172 ms/op 3.2673 ms/op 0.77
Array 250000 clone - spread 1.2244 ms/op 1.2842 ms/op 0.95
Array 250000 get(125000) 0.60000 ns/op 0.65000 ns/op 0.92
Array 250000 set(125000) 0.67600 ns/op 0.71500 ns/op 0.95
Array 250000 iterate all - loop 88.680 us/op 111.25 us/op 0.80
effectiveBalanceIncrements clone Uint8Array 300000 27.697 us/op 41.559 us/op 0.67
effectiveBalanceIncrements clone MutableVector 300000 406.00 ns/op 393.00 ns/op 1.03
effectiveBalanceIncrements rw all Uint8Array 300000 167.47 us/op 187.95 us/op 0.89
effectiveBalanceIncrements rw all MutableVector 300000 83.308 ms/op 92.542 ms/op 0.90
phase0 afterProcessEpoch - 250000 vs - 7PWei 118.65 ms/op 119.75 ms/op 0.99
phase0 beforeProcessEpoch - 250000 vs - 7PWei 41.173 ms/op 47.232 ms/op 0.87
altair processEpoch - mainnet_e81889 330.92 ms/op 351.68 ms/op 0.94
mainnet_e81889 - altair beforeProcessEpoch 61.478 ms/op 98.070 ms/op 0.63
mainnet_e81889 - altair processJustificationAndFinalization 16.249 us/op 19.424 us/op 0.84
mainnet_e81889 - altair processInactivityUpdates 5.6005 ms/op 7.7607 ms/op 0.72
mainnet_e81889 - altair processRewardsAndPenalties 64.483 ms/op 65.076 ms/op 0.99
mainnet_e81889 - altair processRegistryUpdates 2.9290 us/op 3.1920 us/op 0.92
mainnet_e81889 - altair processSlashings 472.00 ns/op 579.00 ns/op 0.82
mainnet_e81889 - altair processEth1DataReset 463.00 ns/op 618.00 ns/op 0.75
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2556 ms/op 1.5891 ms/op 0.79
mainnet_e81889 - altair processSlashingsReset 4.3190 us/op 9.3850 us/op 0.46
mainnet_e81889 - altair processRandaoMixesReset 5.1590 us/op 4.9850 us/op 1.03
mainnet_e81889 - altair processHistoricalRootsUpdate 696.00 ns/op 675.00 ns/op 1.03
mainnet_e81889 - altair processParticipationFlagUpdates 2.6930 us/op 2.6420 us/op 1.02
mainnet_e81889 - altair processSyncCommitteeUpdates 594.00 ns/op 577.00 ns/op 1.03
mainnet_e81889 - altair afterProcessEpoch 125.04 ms/op 132.99 ms/op 0.94
phase0 processEpoch - mainnet_e58758 362.98 ms/op 407.93 ms/op 0.89
mainnet_e58758 - phase0 beforeProcessEpoch 132.71 ms/op 186.32 ms/op 0.71
mainnet_e58758 - phase0 processJustificationAndFinalization 15.975 us/op 24.382 us/op 0.66
mainnet_e58758 - phase0 processRewardsAndPenalties 62.909 ms/op 67.289 ms/op 0.93
mainnet_e58758 - phase0 processRegistryUpdates 7.5260 us/op 8.7610 us/op 0.86
mainnet_e58758 - phase0 processSlashings 482.00 ns/op 577.00 ns/op 0.84
mainnet_e58758 - phase0 processEth1DataReset 535.00 ns/op 604.00 ns/op 0.89
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.2272 ms/op 1.3341 ms/op 0.92
mainnet_e58758 - phase0 processSlashingsReset 4.5020 us/op 3.5800 us/op 1.26
mainnet_e58758 - phase0 processRandaoMixesReset 4.9620 us/op 4.9080 us/op 1.01
mainnet_e58758 - phase0 processHistoricalRootsUpdate 530.00 ns/op 677.00 ns/op 0.78
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.0400 us/op 4.1640 us/op 0.97
mainnet_e58758 - phase0 afterProcessEpoch 96.107 ms/op 113.02 ms/op 0.85
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2171 ms/op 1.3345 ms/op 0.91
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4432 ms/op 2.9214 ms/op 0.49
altair processInactivityUpdates - 250000 normalcase 19.586 ms/op 26.146 ms/op 0.75
altair processInactivityUpdates - 250000 worstcase 18.760 ms/op 28.046 ms/op 0.67
phase0 processRegistryUpdates - 250000 normalcase 7.5500 us/op 5.9500 us/op 1.27
phase0 processRegistryUpdates - 250000 badcase_full_deposits 238.37 us/op 280.20 us/op 0.85
phase0 processRegistryUpdates - 250000 worstcase 0.5 130.43 ms/op 141.47 ms/op 0.92
altair processRewardsAndPenalties - 250000 normalcase 69.356 ms/op 50.839 ms/op 1.36
altair processRewardsAndPenalties - 250000 worstcase 70.579 ms/op 65.531 ms/op 1.08
phase0 getAttestationDeltas - 250000 normalcase 6.9647 ms/op 9.0786 ms/op 0.77
phase0 getAttestationDeltas - 250000 worstcase 6.9499 ms/op 9.5845 ms/op 0.73
phase0 processSlashings - 250000 worstcase 3.6843 ms/op 3.9501 ms/op 0.93
altair processSyncCommitteeUpdates - 250000 177.07 ms/op 200.07 ms/op 0.89
BeaconState.hashTreeRoot - No change 351.00 ns/op 317.00 ns/op 1.11
BeaconState.hashTreeRoot - 1 full validator 49.490 us/op 61.446 us/op 0.81
BeaconState.hashTreeRoot - 32 full validator 541.92 us/op 523.29 us/op 1.04
BeaconState.hashTreeRoot - 512 full validator 5.4831 ms/op 6.4524 ms/op 0.85
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 62.041 us/op 69.562 us/op 0.89
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 997.04 us/op 1.0565 ms/op 0.94
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 12.911 ms/op 14.155 ms/op 0.91
BeaconState.hashTreeRoot - 1 balances 50.902 us/op 54.902 us/op 0.93
BeaconState.hashTreeRoot - 32 balances 499.96 us/op 547.36 us/op 0.91
BeaconState.hashTreeRoot - 512 balances 4.5351 ms/op 5.3989 ms/op 0.84
BeaconState.hashTreeRoot - 250000 balances 75.053 ms/op 81.249 ms/op 0.92
aggregationBits - 2048 els - zipIndexesInBitList 16.684 us/op 17.255 us/op 0.97
regular array get 100000 times 33.577 us/op 34.345 us/op 0.98
wrappedArray get 100000 times 33.557 us/op 44.533 us/op 0.75
arrayWithProxy get 100000 times 17.093 ms/op 15.953 ms/op 1.07
ssz.Root.equals 588.00 ns/op 565.00 ns/op 1.04
byteArrayEquals 575.00 ns/op 565.00 ns/op 1.02
shuffle list - 16384 els 7.0890 ms/op 7.2783 ms/op 0.97
shuffle list - 250000 els 104.79 ms/op 106.49 ms/op 0.98
processSlot - 1 slots 9.1430 us/op 9.6920 us/op 0.94
processSlot - 32 slots 1.4734 ms/op 1.5472 ms/op 0.95
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 37.200 ms/op 37.733 ms/op 0.99
getCommitteeAssignments - req 1 vs - 250000 vc 2.9194 ms/op 3.0751 ms/op 0.95
getCommitteeAssignments - req 100 vs - 250000 vc 4.2042 ms/op 4.3772 ms/op 0.96
getCommitteeAssignments - req 1000 vs - 250000 vc 4.6079 ms/op 4.7543 ms/op 0.97
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.0500 ns/op 4.4300 ns/op 1.14
state getBlockRootAtSlot - 250000 vs - 7PWei 582.28 ns/op 914.22 ns/op 0.64
computeProposers - vc 250000 10.527 ms/op 12.439 ms/op 0.85
computeEpochShuffling - vc 250000 111.50 ms/op 111.59 ms/op 1.00
getNextSyncCommittee - vc 250000 171.01 ms/op 176.22 ms/op 0.97
computeSigningRoot for AttestationData 14.236 us/op 14.273 us/op 1.00
hash AttestationData serialized data then Buffer.toString(base64) 2.5012 us/op 2.6646 us/op 0.94
toHexString serialized data 1.1159 us/op 1.5501 us/op 0.72
Buffer.toString(base64) 344.07 ns/op 386.11 ns/op 0.89

by benchmarkbot/action

@g11tech g11tech enabled auto-merge (squash) April 28, 2023 11:15
@g11tech g11tech force-pushed the g11tech/update-builder-sim branch from 8cfc530 to 8fa11b3 Compare May 1, 2023 14:06
dapplion
dapplion reviewed May 2, 2023
View reviewed changes
packages/beacon-node/test/sim/mergemock.test.ts Outdated
await new Promise<void>((resolve, _reject) => {
bn.chain.emitter.on(routes.events.EventType.block, async (blockData) => {
const {data: fullOrBlindedBlock} = (await bn.api.beacon.getBlockV2(blockData.block).catch((_e) => null)) ?? {};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there's a block event there must be a block available in the API right? This catch all to null seems dangerous

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is to just update the stats, but nevertheless removed the catch 👍

@g11tech g11tech force-pushed the g11tech/update-builder-sim branch from b49d50f to 6dd830e Compare May 2, 2023 05:20
@g11tech g11tech merged commit 597996a into unstable May 4, 2023
11 checks passed
@g11tech g11tech deleted the g11tech/update-builder-sim branch May 4, 2023 03:23
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.9.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@g11tech @wemeetagain @dapplion