Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: skip serializing gossip block when persisting to db #5426

Merged
merged 3 commits into from May 8, 2023
Merged

feat: skip serializing gossip block when persisting to db #5426

merged 3 commits into from May 8, 2023

Conversation

twoeths
Copy link
Contributor

@twoeths twoeths commented Apr 27, 2023

Motivation

Right now when gossipsub deliver a gossip block, we already have serialized data but still have to serialize signed block when persisting to db

Description

  • Add utility types WithBytes, WithOptionalBytes
  • chain now process WithOptionalBytes<BlockInput>, at gossip we pass in bytes data, for blocks from other flow we keep them as is

Closes #5422

@twoeths twoeths changed the title Tuyen/do not serialize gossip blocks feat: skip serializing gossip block when persisting to db Apr 27, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Apr 27, 2023 •
edited

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 547030c Previous: c4f5f05 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 717.21 us/op 754.03 us/op 0.95
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 42.632 us/op 57.642 us/op 0.74
BLS verify - blst-native 1.1591 ms/op 1.2706 ms/op 0.91
BLS verifyMultipleSignatures 3 - blst-native 2.3550 ms/op 2.5771 ms/op 0.91
BLS verifyMultipleSignatures 8 - blst-native 4.9930 ms/op 5.5473 ms/op 0.90
BLS verifyMultipleSignatures 32 - blst-native 18.112 ms/op 20.169 ms/op 0.90
BLS aggregatePubkeys 32 - blst-native 24.203 us/op 27.011 us/op 0.90
BLS aggregatePubkeys 128 - blst-native 94.846 us/op 105.59 us/op 0.90
getAttestationsForBlock 49.632 ms/op 66.165 ms/op 0.75
isKnown best case - 1 super set check 249.00 ns/op 276.00 ns/op 0.90
isKnown normal case - 2 super set checks 241.00 ns/op 269.00 ns/op 0.90
isKnown worse case - 16 super set checks 245.00 ns/op 272.00 ns/op 0.90
CheckpointStateCache - add get delete 4.6740 us/op 5.8660 us/op 0.80
validate gossip signedAggregateAndProof - struct 2.6458 ms/op 2.9038 ms/op 0.91
validate gossip attestation - struct 1.2585 ms/op 1.3965 ms/op 0.90
pickEth1Vote - no votes 1.2122 ms/op 1.3988 ms/op 0.87
pickEth1Vote - max votes 10.011 ms/op 12.308 ms/op 0.81
pickEth1Vote - Eth1Data hashTreeRoot value x2048 8.6114 ms/op 10.493 ms/op 0.82
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 14.939 ms/op 22.242 ms/op 0.67
pickEth1Vote - Eth1Data fastSerialize value x2048 622.80 us/op 808.32 us/op 0.77
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.3726 ms/op 10.338 ms/op 0.62
bytes32 toHexString 475.00 ns/op 789.00 ns/op 0.60
bytes32 Buffer.toString(hex) 338.00 ns/op 479.00 ns/op 0.71
bytes32 Buffer.toString(hex) from Uint8Array 544.00 ns/op 664.00 ns/op 0.82
bytes32 Buffer.toString(hex) + 0x 343.00 ns/op 460.00 ns/op 0.75
Object access 1 prop 0.16100 ns/op 0.22300 ns/op 0.72
Map access 1 prop 0.16200 ns/op 0.17800 ns/op 0.91
Object get x1000 6.5670 ns/op 7.3580 ns/op 0.89
Map get x1000 0.59700 ns/op 0.70300 ns/op 0.85
Object set x1000 51.441 ns/op 79.090 ns/op 0.65
Map set x1000 42.410 ns/op 55.197 ns/op 0.77
Return object 10000 times 0.23200 ns/op 0.25750 ns/op 0.90
Throw Error 10000 times 4.1316 us/op 4.4372 us/op 0.93
fastMsgIdFn sha256 / 200 bytes 3.3060 us/op 3.7060 us/op 0.89
fastMsgIdFn h32 xxhash / 200 bytes 279.00 ns/op 349.00 ns/op 0.80
fastMsgIdFn h64 xxhash / 200 bytes 388.00 ns/op 500.00 ns/op 0.78
fastMsgIdFn sha256 / 1000 bytes 11.194 us/op 12.298 us/op 0.91
fastMsgIdFn h32 xxhash / 1000 bytes 403.00 ns/op 474.00 ns/op 0.85
fastMsgIdFn h64 xxhash / 1000 bytes 463.00 ns/op 593.00 ns/op 0.78
fastMsgIdFn sha256 / 10000 bytes 100.88 us/op 113.46 us/op 0.89
fastMsgIdFn h32 xxhash / 10000 bytes 1.8630 us/op 2.0890 us/op 0.89
fastMsgIdFn h64 xxhash / 10000 bytes 1.3640 us/op 1.4750 us/op 0.92
enrSubnets - fastDeserialize 64 bits 1.3090 us/op 1.8830 us/op 0.70
enrSubnets - ssz BitVector 64 bits 475.00 ns/op 625.00 ns/op 0.76
enrSubnets - fastDeserialize 4 bits 166.00 ns/op 219.00 ns/op 0.76
enrSubnets - ssz BitVector 4 bits 480.00 ns/op 647.00 ns/op 0.74
prioritizePeers score -10:0 att 32-0.1 sync 2-0 103.56 us/op 120.92 us/op 0.86
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 132.54 us/op 157.56 us/op 0.84
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 163.51 us/op 207.92 us/op 0.79
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 295.71 us/op 383.24 us/op 0.77
prioritizePeers score 0:0 att 64-1 sync 4-1 357.66 us/op 450.44 us/op 0.79
array of 16000 items push then shift 1.6042 us/op 1.7264 us/op 0.93
LinkedList of 16000 items push then shift 8.6780 ns/op 9.5670 ns/op 0.91
array of 16000 items push then pop 69.661 ns/op 121.97 ns/op 0.57
LinkedList of 16000 items push then pop 8.2840 ns/op 9.1470 ns/op 0.91
array of 24000 items push then shift 2.3120 us/op 2.4980 us/op 0.93
LinkedList of 24000 items push then shift 8.6610 ns/op 9.6520 ns/op 0.90
array of 24000 items push then pop 72.813 ns/op 90.201 ns/op 0.81
LinkedList of 24000 items push then pop 8.3720 ns/op 9.6310 ns/op 0.87
intersect bitArray bitLen 8 13.024 ns/op 14.603 ns/op 0.89
intersect array and set length 8 76.383 ns/op 90.199 ns/op 0.85
intersect bitArray bitLen 128 43.365 ns/op 45.894 ns/op 0.94
intersect array and set length 128 1.0348 us/op 1.2808 us/op 0.81
Buffer.concat 32 items 2.8160 us/op 3.1200 us/op 0.90
Uint8Array.set 32 items 2.3910 us/op 2.4080 us/op 0.99
pass gossip attestations to forkchoice per slot 2.7193 ms/op 3.4576 ms/op 0.79
computeDeltas 3.0125 ms/op 3.2622 ms/op 0.92
computeProposerBoostScoreFromBalances 1.7764 ms/op 1.8711 ms/op 0.95
altair processAttestation - 250000 vs - 7PWei normalcase 2.5264 ms/op 3.4521 ms/op 0.73
altair processAttestation - 250000 vs - 7PWei worstcase 4.1316 ms/op 4.8280 ms/op 0.86
altair processAttestation - setStatus - 1/6 committees join 140.60 us/op 145.90 us/op 0.96
altair processAttestation - setStatus - 1/3 committees join 271.45 us/op 295.01 us/op 0.92
altair processAttestation - setStatus - 1/2 committees join 360.44 us/op 387.18 us/op 0.93
altair processAttestation - setStatus - 2/3 committees join 460.35 us/op 505.07 us/op 0.91
altair processAttestation - setStatus - 4/5 committees join 645.41 us/op 701.65 us/op 0.92
altair processAttestation - setStatus - 100% committees join 764.22 us/op 808.78 us/op 0.94
altair processBlock - 250000 vs - 7PWei normalcase 15.189 ms/op 19.469 ms/op 0.78
altair processBlock - 250000 vs - 7PWei normalcase hashState 26.840 ms/op 30.325 ms/op 0.89
altair processBlock - 250000 vs - 7PWei worstcase 51.066 ms/op 57.180 ms/op 0.89
altair processBlock - 250000 vs - 7PWei worstcase hashState 71.604 ms/op 77.276 ms/op 0.93
phase0 processBlock - 250000 vs - 7PWei normalcase 2.3770 ms/op 2.4259 ms/op 0.98
phase0 processBlock - 250000 vs - 7PWei worstcase 30.546 ms/op 32.603 ms/op 0.94
altair processEth1Data - 250000 vs - 7PWei normalcase 510.37 us/op 665.43 us/op 0.77
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 11.480 us/op 10.680 us/op 1.07
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 40.256 us/op 32.637 us/op 1.23
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 13.136 us/op 12.620 us/op 1.04
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 9.1570 us/op 9.9820 us/op 0.92
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 108.38 us/op 114.98 us/op 0.94
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 1.0125 ms/op 715.99 us/op 1.41
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.1397 ms/op 921.42 us/op 1.24
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 974.63 us/op 927.25 us/op 1.05
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 2.3615 ms/op 2.9928 ms/op 0.79
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 1.5545 ms/op 1.5713 ms/op 0.99
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 5.0634 ms/op 5.8192 ms/op 0.87
Tree 40 250000 create 391.20 ms/op 473.20 ms/op 0.83
Tree 40 250000 get(125000) 203.84 ns/op 200.97 ns/op 1.01
Tree 40 250000 set(125000) 1.1968 us/op 1.1292 us/op 1.06
Tree 40 250000 toArray() 21.734 ms/op 24.252 ms/op 0.90
Tree 40 250000 iterate all - toArray() + loop 21.963 ms/op 24.455 ms/op 0.90
Tree 40 250000 iterate all - get(i) 74.501 ms/op 79.394 ms/op 0.94
MutableVector 250000 create 11.108 ms/op 13.248 ms/op 0.84
MutableVector 250000 get(125000) 6.5730 ns/op 6.9400 ns/op 0.95
MutableVector 250000 set(125000) 322.02 ns/op 276.17 ns/op 1.17
MutableVector 250000 toArray() 3.5780 ms/op 4.0876 ms/op 0.88
MutableVector 250000 iterate all - toArray() + loop 3.4005 ms/op 4.1445 ms/op 0.82
MutableVector 250000 iterate all - get(i) 1.5592 ms/op 1.5733 ms/op 0.99
Array 250000 create 3.4211 ms/op 3.8693 ms/op 0.88
Array 250000 clone - spread 1.1315 ms/op 1.2424 ms/op 0.91
Array 250000 get(125000) 0.55600 ns/op 0.66300 ns/op 0.84
Array 250000 set(125000) 0.63800 ns/op 0.74500 ns/op 0.86
Array 250000 iterate all - loop 85.131 us/op 86.362 us/op 0.99
effectiveBalanceIncrements clone Uint8Array 300000 29.656 us/op 51.313 us/op 0.58
effectiveBalanceIncrements clone MutableVector 300000 334.00 ns/op 383.00 ns/op 0.87
effectiveBalanceIncrements rw all Uint8Array 300000 165.71 us/op 174.95 us/op 0.95
effectiveBalanceIncrements rw all MutableVector 300000 79.013 ms/op 99.326 ms/op 0.80
phase0 afterProcessEpoch - 250000 vs - 7PWei 114.89 ms/op 121.60 ms/op 0.94
phase0 beforeProcessEpoch - 250000 vs - 7PWei 41.964 ms/op 52.956 ms/op 0.79
altair processEpoch - mainnet_e81889 315.96 ms/op 352.10 ms/op 0.90
mainnet_e81889 - altair beforeProcessEpoch 60.095 ms/op 74.786 ms/op 0.80
mainnet_e81889 - altair processJustificationAndFinalization 19.829 us/op 17.395 us/op 1.14
mainnet_e81889 - altair processInactivityUpdates 7.5300 ms/op 6.0085 ms/op 1.25
mainnet_e81889 - altair processRewardsAndPenalties 71.829 ms/op 63.201 ms/op 1.14
mainnet_e81889 - altair processRegistryUpdates 3.0210 us/op 4.0920 us/op 0.74
mainnet_e81889 - altair processSlashings 521.00 ns/op 584.00 ns/op 0.89
mainnet_e81889 - altair processEth1DataReset 579.00 ns/op 758.00 ns/op 0.76
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.3487 ms/op 1.2870 ms/op 1.05
mainnet_e81889 - altair processSlashingsReset 4.6260 us/op 5.7930 us/op 0.80
mainnet_e81889 - altair processRandaoMixesReset 4.8250 us/op 8.4430 us/op 0.57
mainnet_e81889 - altair processHistoricalRootsUpdate 685.00 ns/op 1.5330 us/op 0.45
mainnet_e81889 - altair processParticipationFlagUpdates 3.7130 us/op 3.6160 us/op 1.03
mainnet_e81889 - altair processSyncCommitteeUpdates 1.0130 us/op 1.0290 us/op 0.98
mainnet_e81889 - altair afterProcessEpoch 130.64 ms/op 131.77 ms/op 0.99
phase0 processEpoch - mainnet_e58758 360.77 ms/op 387.28 ms/op 0.93
mainnet_e58758 - phase0 beforeProcessEpoch 138.74 ms/op 127.90 ms/op 1.08
mainnet_e58758 - phase0 processJustificationAndFinalization 19.993 us/op 19.257 us/op 1.04
mainnet_e58758 - phase0 processRewardsAndPenalties 61.794 ms/op 65.042 ms/op 0.95
mainnet_e58758 - phase0 processRegistryUpdates 7.9970 us/op 10.199 us/op 0.78
mainnet_e58758 - phase0 processSlashings 546.00 ns/op 772.00 ns/op 0.71
mainnet_e58758 - phase0 processEth1DataReset 533.00 ns/op 762.00 ns/op 0.70
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 963.21 us/op 1.0704 ms/op 0.90
mainnet_e58758 - phase0 processSlashingsReset 3.7190 us/op 4.5090 us/op 0.82
mainnet_e58758 - phase0 processRandaoMixesReset 4.2820 us/op 5.5930 us/op 0.77
mainnet_e58758 - phase0 processHistoricalRootsUpdate 548.00 ns/op 787.00 ns/op 0.70
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.5260 us/op 4.2950 us/op 1.05
mainnet_e58758 - phase0 afterProcessEpoch 99.296 ms/op 101.11 ms/op 0.98
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2387 ms/op 1.2809 ms/op 0.97
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.5835 ms/op 1.6275 ms/op 0.97
altair processInactivityUpdates - 250000 normalcase 21.183 ms/op 26.589 ms/op 0.80
altair processInactivityUpdates - 250000 worstcase 22.439 ms/op 25.824 ms/op 0.87
phase0 processRegistryUpdates - 250000 normalcase 6.7650 us/op 8.3950 us/op 0.81
phase0 processRegistryUpdates - 250000 badcase_full_deposits 233.06 us/op 301.03 us/op 0.77
phase0 processRegistryUpdates - 250000 worstcase 0.5 112.79 ms/op 121.84 ms/op 0.93
altair processRewardsAndPenalties - 250000 normalcase 48.146 ms/op 70.190 ms/op 0.69
altair processRewardsAndPenalties - 250000 worstcase 51.879 ms/op 68.454 ms/op 0.76
phase0 getAttestationDeltas - 250000 normalcase 6.6033 ms/op 6.6989 ms/op 0.99
phase0 getAttestationDeltas - 250000 worstcase 7.2285 ms/op 6.9505 ms/op 1.04
phase0 processSlashings - 250000 worstcase 3.6712 ms/op 3.7381 ms/op 0.98
altair processSyncCommitteeUpdates - 250000 187.65 ms/op 179.75 ms/op 1.04
BeaconState.hashTreeRoot - No change 273.00 ns/op 309.00 ns/op 0.88
BeaconState.hashTreeRoot - 1 full validator 50.089 us/op 55.365 us/op 0.90
BeaconState.hashTreeRoot - 32 full validator 529.96 us/op 515.54 us/op 1.03
BeaconState.hashTreeRoot - 512 full validator 5.3879 ms/op 5.0992 ms/op 1.06
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 61.001 us/op 69.823 us/op 0.87
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 893.46 us/op 884.36 us/op 1.01
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 11.623 ms/op 11.134 ms/op 1.04
BeaconState.hashTreeRoot - 1 balances 47.804 us/op 50.632 us/op 0.94
BeaconState.hashTreeRoot - 32 balances 437.49 us/op 458.93 us/op 0.95
BeaconState.hashTreeRoot - 512 balances 4.1917 ms/op 4.4218 ms/op 0.95
BeaconState.hashTreeRoot - 250000 balances 73.428 ms/op 75.272 ms/op 0.98
aggregationBits - 2048 els - zipIndexesInBitList 17.840 us/op 16.717 us/op 1.07
regular array get 100000 times 33.579 us/op 33.470 us/op 1.00
wrappedArray get 100000 times 33.108 us/op 33.886 us/op 0.98
arrayWithProxy get 100000 times 16.655 ms/op 16.155 ms/op 1.03
ssz.Root.equals 572.00 ns/op 588.00 ns/op 0.97
byteArrayEquals 557.00 ns/op 577.00 ns/op 0.97
shuffle list - 16384 els 6.9183 ms/op 7.1900 ms/op 0.96
shuffle list - 250000 els 100.75 ms/op 107.33 ms/op 0.94
processSlot - 1 slots 8.6130 us/op 9.2830 us/op 0.93
processSlot - 32 slots 1.3127 ms/op 1.4533 ms/op 0.90
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 31.658 ms/op 38.425 ms/op 0.82
getCommitteeAssignments - req 1 vs - 250000 vc 2.7992 ms/op 3.0020 ms/op 0.93
getCommitteeAssignments - req 100 vs - 250000 vc 3.9660 ms/op 4.2973 ms/op 0.92
getCommitteeAssignments - req 1000 vs - 250000 vc 4.4393 ms/op 4.6193 ms/op 0.96
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.6300 ns/op 5.3300 ns/op 0.87
state getBlockRootAtSlot - 250000 vs - 7PWei 901.12 ns/op 745.32 ns/op 1.21
computeProposers - vc 250000 11.293 ms/op 11.584 ms/op 0.97
computeEpochShuffling - vc 250000 102.68 ms/op 110.60 ms/op 0.93
getNextSyncCommittee - vc 250000 170.77 ms/op 182.52 ms/op 0.94
computeSigningRoot for AttestationData 13.547 us/op 14.355 us/op 0.94
hash AttestationData serialized data then Buffer.toString(base64) 2.3975 us/op 2.6312 us/op 0.91
toHexString serialized data 1.0846 us/op 1.4499 us/op 0.75
Buffer.toString(base64) 326.26 ns/op 373.00 ns/op 0.87

by benchmarkbot/action

@twoeths twoeths force-pushed the tuyen/do_not_serialize_gossip_blocks branch from 7416368 to b05b060 Compare May 3, 2023 08:41
@twoeths twoeths marked this pull request as ready for review May 3, 2023 09:20
@twoeths twoeths requested a review from a team as a code owner May 3, 2023 09:20
wemeetagain
wemeetagain previously approved these changes May 5, 2023
View reviewed changes
Copy link
Member

@wemeetagain wemeetagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dapplion dapplion merged commit 7c2726a into unstable May 8, 2023
11 checks passed
@dapplion dapplion deleted the tuyen/do_not_serialize_gossip_blocks branch May 8, 2023 03:01
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.9.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

@wemeetagain wemeetagain wemeetagain left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Do not serialize gossip block when persisting to db
3 participants
@twoeths @wemeetagain @dapplion