Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Checkpoint sync only if db state is stale #4563

Merged
merged 1 commit into from Sep 19, 2022
Merged

Checkpoint sync only if db state is stale #4563

merged 1 commit into from Sep 19, 2022

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Sep 19, 2022

Users expects to checkpoint sync only the first time if their db is blank even if they retain the checkpoint args in their setups.

This PR picks the db state to start beacon off if its not stale (within weak subjectivity period). Also improves some logging UX

UX loggin scenarios

Starting from stale db WITHOUT checkpoint sync urls

Sep-18 20:24:50.141[]                 info: Connected to LevelDB database path=/data/testme/chain-db
Sep-18 20:24:50.896[]                 warn: Initializing from a stale db state vulnerable to long range attacks slot=0, epoch=0, stateRoot=0x7e76880eb67bbdc86250aa578958e9d0675e64e714337855204fb5abaaf82c2b, isWithinWeakSubjectivityPeriod=false
Sep-18 20:24:50.897[]                 warn: Checkpoint sync recommended, please use --help to see checkpoint sync options

starting from valid db WITH/WITHOUT checkpoint sync urls

Sep-18 20:23:51.199[]                 info: Lodestar network=mainnet, version=v1.0.0, commit=
Sep-18 20:23:51.838[]                 info: Connected to LevelDB database path=/data/testme/chain-db
Sep-18 20:24:00.692[]                 info: Initializing beacon from a valid db state slot=4725632, epoch=147676, stateRoot=0xd1f9b63264cdfe7582274834a88ad5702f1d12b79c9d1ab2791776d6bf4f79fd, isWithinWeakSubjectivityPeriod=true

starting from a stale/blank db WITH checkpoint sync urls

Sep-18 20:23:10.635[]                 info: Connected to LevelDB database path=/data/testme/chain-db
Sep-18 20:23:10.637[]                 info: Fetching checkpoint state checkpointSyncUrl=https://mainnet-checkpoint-sync.stakely.io
Sep-18 20:23:18.598[]                 info: Download completed
Sep-18 20:23:27.387[]                 info: Initializing beacon from a valid checkpoint state slot=4725632, epoch=147676, stateRoot=0xd1f9b63264cdfe7582274834a88ad5702f1d12b79c9d1ab2791776d6bf4f79fd, isWithinWeakSubjectivityPeriod=true

@g11tech g11tech requested a review from a team as a code owner September 19, 2022 06:49
@github-actions
Copy link
Contributor

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: d17c31b Previous: bb5854b Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.9448 ms/op 1.5574 ms/op 1.25
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 73.327 us/op 63.089 us/op 1.16
BLS verify - blst-native 1.8550 ms/op 2.1626 ms/op 0.86
BLS verifyMultipleSignatures 3 - blst-native 3.8049 ms/op 4.4658 ms/op 0.85
BLS verifyMultipleSignatures 8 - blst-native 8.1816 ms/op 9.6570 ms/op 0.85
BLS verifyMultipleSignatures 32 - blst-native 29.657 ms/op 35.095 ms/op 0.85
BLS aggregatePubkeys 32 - blst-native 39.082 us/op 46.616 us/op 0.84
BLS aggregatePubkeys 128 - blst-native 152.75 us/op 182.18 us/op 0.84
getAttestationsForBlock 84.695 ms/op 72.864 ms/op 1.16
isKnown best case - 1 super set check 428.00 ns/op 476.00 ns/op 0.90
isKnown normal case - 2 super set checks 412.00 ns/op 465.00 ns/op 0.89
isKnown worse case - 16 super set checks 411.00 ns/op 464.00 ns/op 0.89
CheckpointStateCache - add get delete 8.6670 us/op 8.5490 us/op 1.01
validate gossip signedAggregateAndProof - struct 4.2799 ms/op 5.0265 ms/op 0.85
validate gossip attestation - struct 2.0374 ms/op 2.3643 ms/op 0.86
pickEth1Vote - no votes 2.2368 ms/op 2.1631 ms/op 1.03
pickEth1Vote - max votes 19.801 ms/op 17.520 ms/op 1.13
pickEth1Vote - Eth1Data hashTreeRoot value x2048 11.259 ms/op 11.856 ms/op 0.95
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 20.057 ms/op 19.078 ms/op 1.05
pickEth1Vote - Eth1Data fastSerialize value x2048 1.5310 ms/op 1.3796 ms/op 1.11
pickEth1Vote - Eth1Data fastSerialize tree x2048 13.275 ms/op 11.423 ms/op 1.16
bytes32 toHexString 1.0310 us/op 893.00 ns/op 1.15
bytes32 Buffer.toString(hex) 702.00 ns/op 728.00 ns/op 0.96
bytes32 Buffer.toString(hex) from Uint8Array 967.00 ns/op 998.00 ns/op 0.97
bytes32 Buffer.toString(hex) + 0x 690.00 ns/op 733.00 ns/op 0.94
Object access 1 prop 0.34200 ns/op 0.32800 ns/op 1.04
Map access 1 prop 0.29300 ns/op 0.30500 ns/op 0.96
Object get x1000 17.826 ns/op 10.869 ns/op 1.64
Map get x1000 0.97700 ns/op 0.92800 ns/op 1.05
Object set x1000 119.46 ns/op 69.173 ns/op 1.73
Map set x1000 69.295 ns/op 45.618 ns/op 1.52
Return object 10000 times 0.36760 ns/op 0.44000 ns/op 0.84
Throw Error 10000 times 5.9674 us/op 5.7930 us/op 1.03
enrSubnets - fastDeserialize 64 bits 2.6430 us/op 2.4590 us/op 1.07
enrSubnets - ssz BitVector 64 bits 747.00 ns/op 768.00 ns/op 0.97
enrSubnets - fastDeserialize 4 bits 396.00 ns/op 358.00 ns/op 1.11
enrSubnets - ssz BitVector 4 bits 748.00 ns/op 758.00 ns/op 0.99
prioritizePeers score -10:0 att 32-0.1 sync 2-0 93.268 us/op 80.098 us/op 1.16
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 126.39 us/op 114.73 us/op 1.10
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 208.21 us/op 188.66 us/op 1.10
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 453.34 us/op 365.88 us/op 1.24
prioritizePeers score 0:0 att 64-1 sync 4-1 459.60 us/op 412.61 us/op 1.11
RateTracker 1000000 limit, 1 obj count per request 186.15 ns/op 175.20 ns/op 1.06
RateTracker 1000000 limit, 2 obj count per request 138.81 ns/op 127.41 ns/op 1.09
RateTracker 1000000 limit, 4 obj count per request 115.21 ns/op 104.19 ns/op 1.11
RateTracker 1000000 limit, 8 obj count per request 106.36 ns/op 92.429 ns/op 1.15
RateTracker with prune 4.4220 us/op 3.7620 us/op 1.18
array of 16000 items push then shift 3.1992 us/op 51.567 us/op 0.06
LinkedList of 16000 items push then shift 18.544 ns/op 12.059 ns/op 1.54
array of 16000 items push then pop 254.83 ns/op 182.70 ns/op 1.39
LinkedList of 16000 items push then pop 17.405 ns/op 11.837 ns/op 1.47
array of 24000 items push then shift 4.5511 us/op 77.325 us/op 0.06
LinkedList of 24000 items push then shift 22.607 ns/op 12.287 ns/op 1.84
array of 24000 items push then pop 211.98 ns/op 193.66 ns/op 1.09
LinkedList of 24000 items push then pop 19.556 ns/op 12.045 ns/op 1.62
intersect bitArray bitLen 8 11.550 ns/op 10.804 ns/op 1.07
intersect array and set length 8 166.40 ns/op 141.53 ns/op 1.18
intersect bitArray bitLen 128 71.989 ns/op 57.814 ns/op 1.25
intersect array and set length 128 2.2343 us/op 1.7956 us/op 1.24
Buffer.concat 32 items 1.9440 ns/op 1.7220 ns/op 1.13
pass gossip attestations to forkchoice per slot 3.5233 ms/op 2.8846 ms/op 1.22
computeDeltas 3.4925 ms/op 3.0629 ms/op 1.14
computeProposerBoostScoreFromBalances 907.90 us/op 804.07 us/op 1.13
altair processAttestation - 250000 vs - 7PWei normalcase 3.5836 ms/op 3.2450 ms/op 1.10
altair processAttestation - 250000 vs - 7PWei worstcase 5.5770 ms/op 4.9716 ms/op 1.12
altair processAttestation - setStatus - 1/6 committees join 213.19 us/op 179.22 us/op 1.19
altair processAttestation - setStatus - 1/3 committees join 409.89 us/op 351.97 us/op 1.16
altair processAttestation - setStatus - 1/2 committees join 568.60 us/op 504.07 us/op 1.13
altair processAttestation - setStatus - 2/3 committees join 734.43 us/op 663.34 us/op 1.11
altair processAttestation - setStatus - 4/5 committees join 1.0138 ms/op 921.71 us/op 1.10
altair processAttestation - setStatus - 100% committees join 1.2039 ms/op 1.1223 ms/op 1.07
altair processBlock - 250000 vs - 7PWei normalcase 28.259 ms/op 25.987 ms/op 1.09
altair processBlock - 250000 vs - 7PWei normalcase hashState 39.529 ms/op 34.747 ms/op 1.14
altair processBlock - 250000 vs - 7PWei worstcase 80.876 ms/op 73.707 ms/op 1.10
altair processBlock - 250000 vs - 7PWei worstcase hashState 97.925 ms/op 100.84 ms/op 0.97
phase0 processBlock - 250000 vs - 7PWei normalcase 3.5427 ms/op 2.9647 ms/op 1.19
phase0 processBlock - 250000 vs - 7PWei worstcase 45.592 ms/op 49.526 ms/op 0.92
altair processEth1Data - 250000 vs - 7PWei normalcase 953.36 us/op 677.49 us/op 1.41
Tree 40 250000 create 833.11 ms/op 666.94 ms/op 1.25
Tree 40 250000 get(125000) 286.75 ns/op 264.32 ns/op 1.08
Tree 40 250000 set(125000) 2.4730 us/op 2.3493 us/op 1.05
Tree 40 250000 toArray() 32.595 ms/op 25.713 ms/op 1.27
Tree 40 250000 iterate all - toArray() + loop 32.638 ms/op 25.471 ms/op 1.28
Tree 40 250000 iterate all - get(i) 110.50 ms/op 115.15 ms/op 0.96
MutableVector 250000 create 14.447 ms/op 11.538 ms/op 1.25
MutableVector 250000 get(125000) 13.165 ns/op 10.632 ns/op 1.24
MutableVector 250000 set(125000) 627.00 ns/op 442.59 ns/op 1.42
MutableVector 250000 toArray() 7.3592 ms/op 5.3757 ms/op 1.37
MutableVector 250000 iterate all - toArray() + loop 7.3960 ms/op 5.4259 ms/op 1.36
MutableVector 250000 iterate all - get(i) 3.6648 ms/op 2.7192 ms/op 1.35
Array 250000 create 6.0669 ms/op 5.2893 ms/op 1.15
Array 250000 clone - spread 2.6093 ms/op 2.7298 ms/op 0.96
Array 250000 get(125000) 1.0970 ns/op 1.3060 ns/op 0.84
Array 250000 set(125000) 1.1090 ns/op 1.3020 ns/op 0.85
Array 250000 iterate all - loop 167.82 us/op 152.57 us/op 1.10
effectiveBalanceIncrements clone Uint8Array 300000 66.201 us/op 48.649 us/op 1.36
effectiveBalanceIncrements clone MutableVector 300000 652.00 ns/op 627.00 ns/op 1.04
effectiveBalanceIncrements rw all Uint8Array 300000 252.54 us/op 247.12 us/op 1.02
effectiveBalanceIncrements rw all MutableVector 300000 162.82 ms/op 124.08 ms/op 1.31
phase0 afterProcessEpoch - 250000 vs - 7PWei 198.43 ms/op 194.93 ms/op 1.02
phase0 beforeProcessEpoch - 250000 vs - 7PWei 77.595 ms/op 57.841 ms/op 1.34
altair processEpoch - mainnet_e81889 591.42 ms/op 541.52 ms/op 1.09
mainnet_e81889 - altair beforeProcessEpoch 148.09 ms/op 124.44 ms/op 1.19
mainnet_e81889 - altair processJustificationAndFinalization 32.030 us/op 31.819 us/op 1.01
mainnet_e81889 - altair processInactivityUpdates 11.202 ms/op 8.6422 ms/op 1.30
mainnet_e81889 - altair processRewardsAndPenalties 91.551 ms/op 76.634 ms/op 1.19
mainnet_e81889 - altair processRegistryUpdates 6.3920 us/op 5.1070 us/op 1.25
mainnet_e81889 - altair processSlashings 954.00 ns/op 1.1760 us/op 0.81
mainnet_e81889 - altair processEth1DataReset 1.5900 us/op 1.3320 us/op 1.19
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.6741 ms/op 2.5649 ms/op 1.04
mainnet_e81889 - altair processSlashingsReset 12.900 us/op 7.6000 us/op 1.70
mainnet_e81889 - altair processRandaoMixesReset 9.5570 us/op 9.0120 us/op 1.06
mainnet_e81889 - altair processHistoricalRootsUpdate 1.0990 us/op 1.0100 us/op 1.09
mainnet_e81889 - altair processParticipationFlagUpdates 4.3870 us/op 5.9050 us/op 0.74
mainnet_e81889 - altair processSyncCommitteeUpdates 675.00 ns/op 4.0180 us/op 0.17
mainnet_e81889 - altair afterProcessEpoch 193.84 ms/op 219.01 ms/op 0.89
phase0 processEpoch - mainnet_e58758 536.57 ms/op 605.27 ms/op 0.89
mainnet_e58758 - phase0 beforeProcessEpoch 238.03 ms/op 226.72 ms/op 1.05
mainnet_e58758 - phase0 processJustificationAndFinalization 27.430 us/op 28.600 us/op 0.96
mainnet_e58758 - phase0 processRewardsAndPenalties 131.12 ms/op 133.13 ms/op 0.98
mainnet_e58758 - phase0 processRegistryUpdates 18.133 us/op 16.103 us/op 1.13
mainnet_e58758 - phase0 processSlashings 1.3240 us/op 1.3380 us/op 0.99
mainnet_e58758 - phase0 processEth1DataReset 1.3700 us/op 1.5770 us/op 0.87
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.2138 ms/op 2.1886 ms/op 1.01
mainnet_e58758 - phase0 processSlashingsReset 7.4240 us/op 8.0060 us/op 0.93
mainnet_e58758 - phase0 processRandaoMixesReset 10.190 us/op 9.9830 us/op 1.02
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.4400 us/op 1.4190 us/op 1.01
mainnet_e58758 - phase0 processParticipationRecordUpdates 7.3730 us/op 7.3260 us/op 1.01
mainnet_e58758 - phase0 afterProcessEpoch 158.71 ms/op 162.42 ms/op 0.98
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.6349 ms/op 2.4461 ms/op 1.08
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 3.6191 ms/op 2.4739 ms/op 1.46
altair processInactivityUpdates - 250000 normalcase 38.946 ms/op 51.865 ms/op 0.75
altair processInactivityUpdates - 250000 worstcase 48.547 ms/op 50.939 ms/op 0.95
phase0 processRegistryUpdates - 250000 normalcase 14.790 us/op 11.709 us/op 1.26
phase0 processRegistryUpdates - 250000 badcase_full_deposits 572.85 us/op 541.04 us/op 1.06
phase0 processRegistryUpdates - 250000 worstcase 0.5 216.49 ms/op 225.70 ms/op 0.96
altair processRewardsAndPenalties - 250000 normalcase 130.77 ms/op 131.60 ms/op 0.99
altair processRewardsAndPenalties - 250000 worstcase 84.799 ms/op 132.00 ms/op 0.64
phase0 getAttestationDeltas - 250000 normalcase 13.363 ms/op 11.256 ms/op 1.19
phase0 getAttestationDeltas - 250000 worstcase 13.447 ms/op 11.724 ms/op 1.15
phase0 processSlashings - 250000 worstcase 5.5006 ms/op 5.2863 ms/op 1.04
altair processSyncCommitteeUpdates - 250000 273.46 ms/op 278.92 ms/op 0.98
BeaconState.hashTreeRoot - No change 491.00 ns/op 752.00 ns/op 0.65
BeaconState.hashTreeRoot - 1 full validator 60.961 us/op 65.953 us/op 0.92
BeaconState.hashTreeRoot - 32 full validator 726.26 us/op 699.84 us/op 1.04
BeaconState.hashTreeRoot - 512 full validator 6.4817 ms/op 8.2385 ms/op 0.79
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 80.682 us/op 99.326 us/op 0.81
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.1745 ms/op 1.2554 ms/op 0.94
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 15.465 ms/op 17.996 ms/op 0.86
BeaconState.hashTreeRoot - 1 balances 60.277 us/op 66.814 us/op 0.90
BeaconState.hashTreeRoot - 32 balances 582.44 us/op 728.24 us/op 0.80
BeaconState.hashTreeRoot - 512 balances 6.5897 ms/op 6.8991 ms/op 0.96
BeaconState.hashTreeRoot - 250000 balances 92.895 ms/op 113.21 ms/op 0.82
aggregationBits - 2048 els - zipIndexesInBitList 33.637 us/op 25.365 us/op 1.33
regular array get 100000 times 67.435 us/op 60.831 us/op 1.11
wrappedArray get 100000 times 67.371 us/op 61.292 us/op 1.10
arrayWithProxy get 100000 times 28.872 ms/op 37.060 ms/op 0.78
ssz.Root.equals 468.00 ns/op 431.00 ns/op 1.09
byteArrayEquals 442.00 ns/op 427.00 ns/op 1.04
shuffle list - 16384 els 11.066 ms/op 11.282 ms/op 0.98
shuffle list - 250000 els 163.23 ms/op 168.74 ms/op 0.97
processSlot - 1 slots 11.709 us/op 14.095 us/op 0.83
processSlot - 32 slots 1.7178 ms/op 1.8810 ms/op 0.91
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 353.47 us/op 358.42 us/op 0.99
getCommitteeAssignments - req 1 vs - 250000 vc 5.2731 ms/op 5.4573 ms/op 0.97
getCommitteeAssignments - req 100 vs - 250000 vc 7.3362 ms/op 7.9936 ms/op 0.92
getCommitteeAssignments - req 1000 vs - 250000 vc 7.7458 ms/op 8.5641 ms/op 0.90
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 9.6400 ns/op 8.3800 ns/op 1.15
state getBlockRootAtSlot - 250000 vs - 7PWei 1.1099 us/op 1.0012 us/op 1.11
computeProposers - vc 250000 16.393 ms/op 16.674 ms/op 0.98
computeEpochShuffling - vc 250000 168.68 ms/op 171.18 ms/op 0.99
getNextSyncCommittee - vc 250000 275.21 ms/op 277.68 ms/op 0.99

by benchmarkbot/action

@dapplion
Copy link
Contributor

Much clearer!

@g11tech g11tech merged commit 639cd41 into unstable Sep 19, 2022
@g11tech g11tech deleted the g11tech/checkpoint-ux branch September 19, 2022 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dapplion dapplion dapplion approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@g11tech @dapplion