Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bump level db to 8.0.0 #4514

Merged
merged 4 commits into from Sep 7, 2022
Merged

Bump level db to 8.0.0 #4514

merged 4 commits into from Sep 7, 2022

Conversation

dapplion
Copy link
Contributor

@dapplion dapplion commented Sep 7, 2022

Motivation

Description

  • Bump level to 8.0.0
  • Drop custom wrapper code for the iterator
  • Use existing .keys() and .values() methods where possible

Functionality should be equivalent, only less wanky code to mantain

@dapplion dapplion requested a review from a team as a code owner September 7, 2022 08:35
@dapplion dapplion changed the title Bump level to 8.0.0 Bump level db to 8.0.0 Sep 7, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Sep 7, 2022 •
edited

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: cbd6c93 Previous: 91a5a06 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 3.0331 ms/op 2.4283 ms/op 1.25
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 95.087 us/op 75.539 us/op 1.26
BLS verify - blst-native 2.4173 ms/op 1.8554 ms/op 1.30
BLS verifyMultipleSignatures 3 - blst-native 4.8484 ms/op 3.8102 ms/op 1.27
BLS verifyMultipleSignatures 8 - blst-native 10.742 ms/op 8.1969 ms/op 1.31
BLS verifyMultipleSignatures 32 - blst-native 38.809 ms/op 29.703 ms/op 1.31
BLS aggregatePubkeys 32 - blst-native 51.502 us/op 39.393 us/op 1.31
BLS aggregatePubkeys 128 - blst-native 202.92 us/op 152.86 us/op 1.33
getAttestationsForBlock 223.20 ms/op 151.09 ms/op 1.48
isKnown best case - 1 super set check 572.00 ns/op 379.00 ns/op 1.51
isKnown normal case - 2 super set checks 530.00 ns/op 372.00 ns/op 1.42
isKnown worse case - 16 super set checks 532.00 ns/op 368.00 ns/op 1.45
CheckpointStateCache - add get delete 11.799 us/op 8.0120 us/op 1.47
validate gossip signedAggregateAndProof - struct 5.6240 ms/op 3.7593 ms/op 1.50
validate gossip attestation - struct 2.6418 ms/op 1.7895 ms/op 1.48
pickEth1Vote - no votes 2.7827 ms/op 1.9817 ms/op 1.40
pickEth1Vote - max votes 27.439 ms/op 19.642 ms/op 1.40
pickEth1Vote - Eth1Data hashTreeRoot value x2048 14.919 ms/op 11.606 ms/op 1.29
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 26.837 ms/op 20.271 ms/op 1.32
pickEth1Vote - Eth1Data fastSerialize value x2048 2.0223 ms/op 1.3884 ms/op 1.46
pickEth1Vote - Eth1Data fastSerialize tree x2048 17.439 ms/op 11.845 ms/op 1.47
bytes32 toHexString 1.3160 us/op 937.00 ns/op 1.40
bytes32 Buffer.toString(hex) 930.00 ns/op 688.00 ns/op 1.35
bytes32 Buffer.toString(hex) from Uint8Array 1.2540 us/op 898.00 ns/op 1.40
bytes32 Buffer.toString(hex) + 0x 971.00 ns/op 691.00 ns/op 1.41
Object access 1 prop 0.52300 ns/op 0.34400 ns/op 1.52
Map access 1 prop 0.39700 ns/op 0.29900 ns/op 1.33
Object get x1000 17.046 ns/op 17.062 ns/op 1.00
Map get x1000 1.1650 ns/op 0.98800 ns/op 1.18
Object set x1000 123.88 ns/op 112.21 ns/op 1.10
Map set x1000 89.704 ns/op 69.292 ns/op 1.29
Return object 10000 times 0.46300 ns/op 0.36980 ns/op 1.25
Throw Error 10000 times 8.6762 us/op 5.8525 us/op 1.48
enrSubnets - fastDeserialize 64 bits 3.5890 us/op 2.5200 us/op 1.42
enrSubnets - ssz BitVector 64 bits 937.00 ns/op 698.00 ns/op 1.34
enrSubnets - fastDeserialize 4 bits 536.00 ns/op 365.00 ns/op 1.47
enrSubnets - ssz BitVector 4 bits 950.00 ns/op 710.00 ns/op 1.34
prioritizePeers score -10:0 att 32-0.1 sync 2-0 125.45 us/op 91.308 us/op 1.37
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 180.82 us/op 141.49 us/op 1.28
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 334.88 us/op 206.69 us/op 1.62
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 584.87 us/op 389.62 us/op 1.50
prioritizePeers score 0:0 att 64-1 sync 4-1 616.80 us/op 460.26 us/op 1.34
RateTracker 1000000 limit, 1 obj count per request 248.64 ns/op 190.79 ns/op 1.30
RateTracker 1000000 limit, 2 obj count per request 179.74 ns/op 147.01 ns/op 1.22
RateTracker 1000000 limit, 4 obj count per request 153.53 ns/op 120.18 ns/op 1.28
RateTracker 1000000 limit, 8 obj count per request 142.20 ns/op 109.98 ns/op 1.29
RateTracker with prune 5.9340 us/op 4.1180 us/op 1.44
array of 16000 items push then shift 5.8854 us/op 3.1847 us/op 1.85
LinkedList of 16000 items push then shift 22.750 ns/op 17.580 ns/op 1.29
array of 16000 items push then pop 294.46 ns/op 233.50 ns/op 1.26
LinkedList of 16000 items push then pop 20.094 ns/op 14.854 ns/op 1.35
array of 24000 items push then shift 8.2805 us/op 4.0148 us/op 2.06
LinkedList of 24000 items push then shift 22.856 ns/op 19.069 ns/op 1.20
array of 24000 items push then pop 270.13 ns/op 188.49 ns/op 1.43
LinkedList of 24000 items push then pop 19.887 ns/op 16.379 ns/op 1.21
intersect bitArray bitLen 8 13.257 ns/op 10.098 ns/op 1.31
intersect array and set length 8 228.68 ns/op 141.33 ns/op 1.62
intersect bitArray bitLen 128 74.978 ns/op 63.463 ns/op 1.18
intersect array and set length 128 2.6220 us/op 1.9281 us/op 1.36
Buffer.concat 32 items 2.9210 ns/op 1.6740 ns/op 1.74
pass gossip attestations to forkchoice per slot 3.8055 ms/op 2.8960 ms/op 1.31
computeDeltas 4.7476 ms/op 2.9197 ms/op 1.63
computeProposerBoostScoreFromBalances 945.66 us/op 804.77 us/op 1.18
altair processAttestation - 250000 vs - 7PWei normalcase 5.9285 ms/op 3.0915 ms/op 1.92
altair processAttestation - 250000 vs - 7PWei worstcase 9.1436 ms/op 4.8030 ms/op 1.90
altair processAttestation - setStatus - 1/6 committees join 259.39 us/op 191.90 us/op 1.35
altair processAttestation - setStatus - 1/3 committees join 518.59 us/op 409.01 us/op 1.27
altair processAttestation - setStatus - 1/2 committees join 737.42 us/op 513.26 us/op 1.44
altair processAttestation - setStatus - 2/3 committees join 1.0068 ms/op 660.84 us/op 1.52
altair processAttestation - setStatus - 4/5 committees join 1.3511 ms/op 928.33 us/op 1.46
altair processAttestation - setStatus - 100% committees join 1.6608 ms/op 1.2349 ms/op 1.34
altair processBlock - 250000 vs - 7PWei normalcase 36.338 ms/op 28.512 ms/op 1.27
altair processBlock - 250000 vs - 7PWei normalcase hashState 45.274 ms/op 43.749 ms/op 1.03
altair processBlock - 250000 vs - 7PWei worstcase 125.00 ms/op 68.806 ms/op 1.82
altair processBlock - 250000 vs - 7PWei worstcase hashState 130.22 ms/op 98.279 ms/op 1.33
phase0 processBlock - 250000 vs - 7PWei normalcase 5.0527 ms/op 3.2878 ms/op 1.54
phase0 processBlock - 250000 vs - 7PWei worstcase 62.115 ms/op 47.652 ms/op 1.30
altair processEth1Data - 250000 vs - 7PWei normalcase 1.1240 ms/op 723.93 us/op 1.55
Tree 40 250000 create 1.1553 s/op 760.88 ms/op 1.52
Tree 40 250000 get(125000) 352.62 ns/op 257.33 ns/op 1.37
Tree 40 250000 set(125000) 4.0989 us/op 2.3515 us/op 1.74
Tree 40 250000 toArray() 41.718 ms/op 29.413 ms/op 1.42
Tree 40 250000 iterate all - toArray() + loop 40.183 ms/op 29.509 ms/op 1.36
Tree 40 250000 iterate all - get(i) 146.34 ms/op 98.359 ms/op 1.49
MutableVector 250000 create 21.056 ms/op 23.734 ms/op 0.89
MutableVector 250000 get(125000) 15.044 ns/op 13.612 ns/op 1.11
MutableVector 250000 set(125000) 1.0853 us/op 478.34 ns/op 2.27
MutableVector 250000 toArray() 9.6035 ms/op 6.6508 ms/op 1.44
MutableVector 250000 iterate all - toArray() + loop 7.6902 ms/op 6.4308 ms/op 1.20
MutableVector 250000 iterate all - get(i) 3.7048 ms/op 3.0369 ms/op 1.22
Array 250000 create 7.4029 ms/op 6.2554 ms/op 1.18
Array 250000 clone - spread 3.3347 ms/op 2.5267 ms/op 1.32
Array 250000 get(125000) 1.4300 ns/op 1.0360 ns/op 1.38
Array 250000 set(125000) 1.5040 ns/op 1.0430 ns/op 1.44
Array 250000 iterate all - loop 160.57 us/op 148.02 us/op 1.08
effectiveBalanceIncrements clone Uint8Array 300000 215.87 us/op 59.158 us/op 3.65
effectiveBalanceIncrements clone MutableVector 300000 826.00 ns/op 719.00 ns/op 1.15
effectiveBalanceIncrements rw all Uint8Array 300000 341.29 us/op 222.66 us/op 1.53
effectiveBalanceIncrements rw all MutableVector 300000 249.84 ms/op 163.20 ms/op 1.53
phase0 afterProcessEpoch - 250000 vs - 7PWei 226.71 ms/op 159.89 ms/op 1.42
phase0 beforeProcessEpoch - 250000 vs - 7PWei 84.339 ms/op 74.089 ms/op 1.14
altair processEpoch - mainnet_e81889 751.60 ms/op 532.97 ms/op 1.41
mainnet_e81889 - altair beforeProcessEpoch 119.07 ms/op 169.07 ms/op 0.70
mainnet_e81889 - altair processJustificationAndFinalization 69.529 us/op 25.980 us/op 2.68
mainnet_e81889 - altair processInactivityUpdates 13.434 ms/op 9.9964 ms/op 1.34
mainnet_e81889 - altair processRewardsAndPenalties 209.70 ms/op 86.141 ms/op 2.43
mainnet_e81889 - altair processRegistryUpdates 14.656 us/op 4.8680 us/op 3.01
mainnet_e81889 - altair processSlashings 4.0250 us/op 1.0100 us/op 3.99
mainnet_e81889 - altair processEth1DataReset 4.2320 us/op 1.0140 us/op 4.17
mainnet_e81889 - altair processEffectiveBalanceUpdates 3.9634 ms/op 2.1081 ms/op 1.88
mainnet_e81889 - altair processSlashingsReset 26.782 us/op 7.1230 us/op 3.76
mainnet_e81889 - altair processRandaoMixesReset 25.982 us/op 9.1900 us/op 2.83
mainnet_e81889 - altair processHistoricalRootsUpdate 3.9720 us/op 1.4050 us/op 2.83
mainnet_e81889 - altair processParticipationFlagUpdates 14.390 us/op 4.1250 us/op 3.49
mainnet_e81889 - altair processSyncCommitteeUpdates 4.0510 us/op 1.6060 us/op 2.52
mainnet_e81889 - altair afterProcessEpoch 219.52 ms/op 218.22 ms/op 1.01
phase0 processEpoch - mainnet_e58758 675.13 ms/op 579.15 ms/op 1.17
mainnet_e58758 - phase0 beforeProcessEpoch 344.23 ms/op 231.33 ms/op 1.49
mainnet_e58758 - phase0 processJustificationAndFinalization 61.162 us/op 32.001 us/op 1.91
mainnet_e58758 - phase0 processRewardsAndPenalties 100.13 ms/op 121.45 ms/op 0.82
mainnet_e58758 - phase0 processRegistryUpdates 34.227 us/op 12.805 us/op 2.67
mainnet_e58758 - phase0 processSlashings 3.4290 us/op 1.0010 us/op 3.43
mainnet_e58758 - phase0 processEth1DataReset 3.1180 us/op 1.3660 us/op 2.28
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.0995 ms/op 2.2824 ms/op 0.92
mainnet_e58758 - phase0 processSlashingsReset 15.829 us/op 8.0700 us/op 1.96
mainnet_e58758 - phase0 processRandaoMixesReset 24.842 us/op 7.5800 us/op 3.28
mainnet_e58758 - phase0 processHistoricalRootsUpdate 3.6980 us/op 1.2880 us/op 2.87
mainnet_e58758 - phase0 processParticipationRecordUpdates 22.095 us/op 7.1590 us/op 3.09
mainnet_e58758 - phase0 afterProcessEpoch 185.08 ms/op 155.59 ms/op 1.19
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.6282 ms/op 2.6932 ms/op 0.98
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 3.7643 ms/op 2.7772 ms/op 1.36
altair processInactivityUpdates - 250000 normalcase 60.088 ms/op 41.106 ms/op 1.46
altair processInactivityUpdates - 250000 worstcase 61.804 ms/op 51.346 ms/op 1.20
phase0 processRegistryUpdates - 250000 normalcase 27.477 us/op 10.535 us/op 2.61
phase0 processRegistryUpdates - 250000 badcase_full_deposits 571.45 us/op 363.21 us/op 1.57
phase0 processRegistryUpdates - 250000 worstcase 0.5 313.38 ms/op 213.92 ms/op 1.46
altair processRewardsAndPenalties - 250000 normalcase 163.21 ms/op 111.53 ms/op 1.46
altair processRewardsAndPenalties - 250000 worstcase 104.70 ms/op 85.889 ms/op 1.22
phase0 getAttestationDeltas - 250000 normalcase 16.968 ms/op 12.191 ms/op 1.39
phase0 getAttestationDeltas - 250000 worstcase 16.480 ms/op 12.630 ms/op 1.30
phase0 processSlashings - 250000 worstcase 6.9917 ms/op 4.7091 ms/op 1.48
altair processSyncCommitteeUpdates - 250000 370.63 ms/op 263.91 ms/op 1.40
BeaconState.hashTreeRoot - No change 644.00 ns/op 412.00 ns/op 1.56
BeaconState.hashTreeRoot - 1 full validator 81.912 us/op 57.071 us/op 1.44
BeaconState.hashTreeRoot - 32 full validator 1.0073 ms/op 642.13 us/op 1.57
BeaconState.hashTreeRoot - 512 full validator 8.1077 ms/op 6.0297 ms/op 1.34
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 101.83 us/op 73.371 us/op 1.39
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.5298 ms/op 1.0951 ms/op 1.40
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 20.452 ms/op 16.822 ms/op 1.22
BeaconState.hashTreeRoot - 1 balances 76.834 us/op 55.174 us/op 1.39
BeaconState.hashTreeRoot - 32 balances 743.93 us/op 549.64 us/op 1.35
BeaconState.hashTreeRoot - 512 balances 7.6424 ms/op 6.5468 ms/op 1.17
BeaconState.hashTreeRoot - 250000 balances 113.56 ms/op 86.179 ms/op 1.32
aggregationBits - 2048 els - zipIndexesInBitList 45.520 us/op 36.947 us/op 1.23
regular array get 100000 times 59.731 us/op 59.520 us/op 1.00
wrappedArray get 100000 times 57.660 us/op 59.493 us/op 0.97
arrayWithProxy get 100000 times 35.050 ms/op 25.780 ms/op 1.36
ssz.Root.equals 666.00 ns/op 480.00 ns/op 1.39
byteArrayEquals 624.00 ns/op 476.00 ns/op 1.31
shuffle list - 16384 els 12.890 ms/op 9.7327 ms/op 1.32
shuffle list - 250000 els 183.14 ms/op 143.56 ms/op 1.28
processSlot - 1 slots 20.567 us/op 13.140 us/op 1.57
processSlot - 32 slots 2.5857 ms/op 1.7704 ms/op 1.46
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 623.78 us/op 390.84 us/op 1.60
getCommitteeAssignments - req 1 vs - 250000 vc 5.7112 ms/op 5.2691 ms/op 1.08
getCommitteeAssignments - req 100 vs - 250000 vc 8.2275 ms/op 7.3471 ms/op 1.12
getCommitteeAssignments - req 1000 vs - 250000 vc 9.1405 ms/op 8.1887 ms/op 1.12
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 11.080 ns/op 10.210 ns/op 1.09
state getBlockRootAtSlot - 250000 vs - 7PWei 1.4232 us/op 1.0192 us/op 1.40
computeProposers - vc 250000 22.349 ms/op 14.500 ms/op 1.54
computeEpochShuffling - vc 250000 200.97 ms/op 169.09 ms/op 1.19
getNextSyncCommittee - vc 250000 376.09 ms/op 282.95 ms/op 1.33

by benchmarkbot/action

twoeths
twoeths reviewed Sep 7, 2022
View reviewed changes
packages/db/test/unit/controller/level.test.ts
await db.compactRange(minKey, maxKey);
const sizeAfterCompact = getDbSize();

expect(sizeAfterCompact).lt(sizeBeforeCompact, "Expected sizeAfterCompact < sizeBeforeCompact");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how much disk space was saved with this test? I think we should expect some threshold here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not that familiar with level internals so I think it's dangeours to do assumptions here. LevelDB behaviour is not trivial

twoeths
twoeths approved these changes Sep 7, 2022
View reviewed changes
Copy link
Contributor

@twoeths twoeths left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@dapplion dapplion merged commit daf2f28 into unstable Sep 7, 2022
@dapplion dapplion deleted the dapplion/bump-level branch September 7, 2022 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@twoeths twoeths twoeths approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@dapplion @twoeths