Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter out files that has data_store in name when resolving keystore path #3634

Merged
merged 7 commits into from
Jan 18, 2022

Conversation

dadepo
Copy link
Contributor

@dadepo dadepo commented Jan 17, 2022

Motivation

Prevents lodestar from crashing if deposit data files are encountered when resolving keystore path.

Description

Filtering out file names that contains deposit_data, when solving keystore path so as to prevent crashing on start up.

Closes #3611

Steps to test or reproduce

  • Put deposit_data.json files in the /keystores folder.
  • Run the validator client with --importKeystoresPath flag
  • Receive object expected error.

@codecov
Copy link

codecov bot commented Jan 17, 2022

Codecov Report

Merging #3634 (93fe3a5) into master (b384249) will decrease coverage by 0.98%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3634      +/-   ##
==========================================
- Coverage   37.50%   36.51%   -0.99%     
==========================================
  Files         311      320       +9     
  Lines        8357     8960     +603     
  Branches     1295     1432     +137     
==========================================
+ Hits         3134     3272     +138     
- Misses       5074     5536     +462     
- Partials      149      152       +3     

@codeclimate
Copy link

codeclimate bot commented Jan 17, 2022

Code Climate has analyzed commit 93fe3a5 and detected 0 issues on this pull request.

View more on Code Climate.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 17, 2022

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 50f8104 Previous: e2c79e4 Ratio
BeaconState.hashTreeRoot - No change 561.00 ns/op 501.00 ns/op 1.12
BeaconState.hashTreeRoot - 1 full validator 128.85 us/op 120.85 us/op 1.07
BeaconState.hashTreeRoot - 32 full validator 1.9414 ms/op 1.8772 ms/op 1.03
BeaconState.hashTreeRoot - 512 full validator 25.806 ms/op 25.443 ms/op 1.01
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 128.97 us/op 121.00 us/op 1.07
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.0697 ms/op 1.9453 ms/op 1.06
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 28.934 ms/op 26.096 ms/op 1.11
BeaconState.hashTreeRoot - 1 balances 90.987 us/op 90.992 us/op 1.00
BeaconState.hashTreeRoot - 32 balances 754.26 us/op 716.88 us/op 1.05
BeaconState.hashTreeRoot - 512 balances 7.4979 ms/op 7.1371 ms/op 1.05
BeaconState.hashTreeRoot - 250000 balances 138.01 ms/op 135.38 ms/op 1.02
processSlot - 1 slots 48.371 us/op 46.218 us/op 1.05
processSlot - 32 slots 2.9188 ms/op 2.9334 ms/op 1.00
getCommitteeAssignments - req 1 vs - 250000 vc 5.3384 ms/op 5.3188 ms/op 1.00
getCommitteeAssignments - req 100 vs - 250000 vc 7.4465 ms/op 7.4765 ms/op 1.00
getCommitteeAssignments - req 1000 vs - 250000 vc 7.9366 ms/op 7.8874 ms/op 1.01
computeProposers - vc 250000 21.376 ms/op 20.918 ms/op 1.02
computeEpochShuffling - vc 250000 189.65 ms/op 190.07 ms/op 1.00
getNextSyncCommittee - vc 250000 350.72 ms/op 344.19 ms/op 1.02
altair processAttestation - 250000 vs - 7PWei normalcase 47.884 ms/op 42.880 ms/op 1.12
altair processAttestation - 250000 vs - 7PWei worstcase 48.016 ms/op 45.919 ms/op 1.05
altair processAttestation - setStatus - 1/6 committees join 11.749 ms/op 10.712 ms/op 1.10
altair processAttestation - setStatus - 1/3 committees join 25.910 ms/op 23.123 ms/op 1.12
altair processAttestation - setStatus - 1/2 committees join 39.040 ms/op 36.589 ms/op 1.07
altair processAttestation - setStatus - 2/3 committees join 53.035 ms/op 46.696 ms/op 1.14
altair processAttestation - setStatus - 4/5 committees join 62.124 ms/op 57.818 ms/op 1.07
altair processAttestation - setStatus - 100% committees join 80.839 ms/op 73.109 ms/op 1.11
altair processAttestation - updateEpochParticipants - 1/6 committees join 13.891 ms/op 12.158 ms/op 1.14
altair processAttestation - updateEpochParticipants - 1/3 committees join 28.798 ms/op 24.469 ms/op 1.18
altair processAttestation - updateEpochParticipants - 1/2 committees join 29.583 ms/op 26.819 ms/op 1.10
altair processAttestation - updateEpochParticipants - 2/3 committees join 36.456 ms/op 64.821 ms/op 0.56
altair processAttestation - updateEpochParticipants - 4/5 committees join 32.800 ms/op 25.873 ms/op 1.27
altair processAttestation - updateEpochParticipants - 100% committees join 33.944 ms/op 28.237 ms/op 1.20
altair processAttestation - updateAllStatus 24.391 ms/op 21.082 ms/op 1.16
altair processBlock - 250000 vs - 7PWei normalcase 51.631 ms/op 45.311 ms/op 1.14
altair processBlock - 250000 vs - 7PWei worstcase 127.39 ms/op 113.91 ms/op 1.12
altair processEpoch - mainnet_e81889 1.0761 s/op 990.96 ms/op 1.09
mainnet_e81889 - altair beforeProcessEpoch 304.58 ms/op 260.61 ms/op 1.17
mainnet_e81889 - altair processJustificationAndFinalization 50.473 us/op 67.737 us/op 0.75
mainnet_e81889 - altair processInactivityUpdates 18.992 ms/op 17.685 ms/op 1.07
mainnet_e81889 - altair processRewardsAndPenalties 149.42 ms/op 143.61 ms/op 1.04
mainnet_e81889 - altair processRegistryUpdates 5.7000 us/op 8.4860 us/op 0.67
mainnet_e81889 - altair processSlashings 1.1560 us/op 1.6230 us/op 0.71
mainnet_e81889 - altair processEth1DataReset 1.0890 us/op 1.6710 us/op 0.65
mainnet_e81889 - altair processEffectiveBalanceUpdates 11.903 ms/op 10.516 ms/op 1.13
mainnet_e81889 - altair processSlashingsReset 8.3930 us/op 13.795 us/op 0.61
mainnet_e81889 - altair processRandaoMixesReset 12.268 us/op 19.888 us/op 0.62
mainnet_e81889 - altair processHistoricalRootsUpdate 1.2740 us/op 2.2330 us/op 0.57
mainnet_e81889 - altair processParticipationFlagUpdates 172.04 ms/op 124.99 ms/op 1.38
mainnet_e81889 - altair processSyncCommitteeUpdates 1.1130 us/op 1.2390 us/op 0.90
mainnet_e81889 - altair afterProcessEpoch 222.68 ms/op 219.77 ms/op 1.01
altair processInactivityUpdates - 250000 normalcase 72.182 ms/op 65.397 ms/op 1.10
altair processInactivityUpdates - 250000 worstcase 74.811 ms/op 73.332 ms/op 1.02
altair processParticipationFlagUpdates - 250000 anycase 97.545 ms/op 91.969 ms/op 1.06
altair processRewardsAndPenalties - 250000 normalcase 134.62 ms/op 122.93 ms/op 1.10
altair processRewardsAndPenalties - 250000 worstcase 153.63 ms/op 145.89 ms/op 1.05
altair processSyncCommitteeUpdates - 250000 373.60 ms/op 325.62 ms/op 1.15
Tree 40 250000 create 919.31 ms/op 814.10 ms/op 1.13
Tree 40 250000 get(125000) 327.45 ns/op 282.01 ns/op 1.16
Tree 40 250000 set(125000) 2.1983 us/op 1.8911 us/op 1.16
Tree 40 250000 toArray() 48.997 ms/op 44.010 ms/op 1.11
Tree 40 250000 iterate all - toArray() + loop 41.604 ms/op 38.047 ms/op 1.09
Tree 40 250000 iterate all - get(i) 121.79 ms/op 120.35 ms/op 1.01
MutableVector 250000 create 26.026 ms/op 20.484 ms/op 1.27
MutableVector 250000 get(125000) 14.229 ns/op 13.968 ns/op 1.02
MutableVector 250000 set(125000) 567.87 ns/op 480.64 ns/op 1.18
MutableVector 250000 toArray() 9.2550 ms/op 14.518 ms/op 0.64
MutableVector 250000 iterate all - toArray() + loop 9.1581 ms/op 8.6375 ms/op 1.06
MutableVector 250000 iterate all - get(i) 3.4194 ms/op 3.1476 ms/op 1.09
Array 250000 create 6.1131 ms/op 5.0720 ms/op 1.21
Array 250000 clone - spread 2.3260 ms/op 1.7980 ms/op 1.29
Array 250000 get(125000) 1.0940 ns/op 0.88400 ns/op 1.24
Array 250000 set(125000) 1.0760 ns/op 0.87400 ns/op 1.23
Array 250000 iterate all - loop 167.82 us/op 167.78 us/op 1.00
aggregationBits - 2048 els - readonlyValues 245.95 us/op 205.91 us/op 1.19
aggregationBits - 2048 els - zipIndexesInBitList 52.230 us/op 37.063 us/op 1.41
regular array get 100000 times 67.422 us/op 67.389 us/op 1.00
wrappedArray get 100000 times 67.457 us/op 67.397 us/op 1.00
arrayWithProxy get 100000 times 28.816 ms/op 28.397 ms/op 1.01
ssz.Root.equals 1.2060 us/op 1.0790 us/op 1.12
ssz.Root.equals with valueOf() 1.5370 us/op 1.2740 us/op 1.21
byteArrayEquals with valueOf() 1.5190 us/op 1.3270 us/op 1.14
phase0 processBlock - 250000 vs - 7PWei normalcase 11.051 ms/op 10.068 ms/op 1.10
phase0 processBlock - 250000 vs - 7PWei worstcase 79.559 ms/op 73.977 ms/op 1.08
phase0 afterProcessEpoch - 250000 vs - 7PWei 208.35 ms/op 189.14 ms/op 1.10
phase0 beforeProcessEpoch - 250000 vs - 7PWei 623.32 ms/op 536.30 ms/op 1.16
phase0 processEpoch - mainnet_e58758 828.40 ms/op 760.58 ms/op 1.09
mainnet_e58758 - phase0 beforeProcessEpoch 453.18 ms/op 462.43 ms/op 0.98
mainnet_e58758 - phase0 processJustificationAndFinalization 59.918 us/op 86.346 us/op 0.69
mainnet_e58758 - phase0 processRewardsAndPenalties 103.95 ms/op 122.83 ms/op 0.85
mainnet_e58758 - phase0 processRegistryUpdates 36.300 us/op 49.800 us/op 0.73
mainnet_e58758 - phase0 processSlashings 1.1500 us/op 1.9130 us/op 0.60
mainnet_e58758 - phase0 processEth1DataReset 1.0010 us/op 1.7280 us/op 0.58
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 9.9422 ms/op 8.3585 ms/op 1.19
mainnet_e58758 - phase0 processSlashingsReset 7.7790 us/op 10.935 us/op 0.71
mainnet_e58758 - phase0 processRandaoMixesReset 10.773 us/op 13.183 us/op 0.82
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.1800 us/op 1.4410 us/op 0.82
mainnet_e58758 - phase0 processParticipationRecordUpdates 8.1530 us/op 11.884 us/op 0.69
mainnet_e58758 - phase0 afterProcessEpoch 182.84 ms/op 165.70 ms/op 1.10
phase0 processEffectiveBalanceUpdates - 250000 normalcase 10.733 ms/op 9.6318 ms/op 1.11
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4799 s/op 1.3132 s/op 1.13
phase0 processRegistryUpdates - 250000 normalcase 47.207 us/op 57.051 us/op 0.83
phase0 processRegistryUpdates - 250000 badcase_full_deposits 3.0214 ms/op 2.7625 ms/op 1.09
phase0 processRegistryUpdates - 250000 worstcase 0.5 1.6232 s/op 1.6858 s/op 0.96
phase0 getAttestationDeltas - 250000 normalcase 35.243 ms/op 35.246 ms/op 1.00
phase0 getAttestationDeltas - 250000 worstcase 35.217 ms/op 35.466 ms/op 0.99
phase0 processSlashings - 250000 worstcase 40.785 ms/op 36.332 ms/op 1.12
shuffle list - 16384 els 12.929 ms/op 12.695 ms/op 1.02
shuffle list - 250000 els 185.39 ms/op 183.07 ms/op 1.01
getEffectiveBalances - 250000 vs - 7PWei 12.059 ms/op 9.4453 ms/op 1.28
pass gossip attestations to forkchoice per slot 16.414 ms/op 14.076 ms/op 1.17
computeDeltas 3.8222 ms/op 3.6642 ms/op 1.04
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.6154 ms/op 2.0446 ms/op 1.28
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 691.98 us/op 566.30 us/op 1.22
BLS verify - blst-native 1.8588 ms/op 1.6355 ms/op 1.14
BLS verifyMultipleSignatures 3 - blst-native 3.8150 ms/op 3.3557 ms/op 1.14
BLS verifyMultipleSignatures 8 - blst-native 8.2403 ms/op 7.2408 ms/op 1.14
BLS verifyMultipleSignatures 32 - blst-native 29.861 ms/op 26.289 ms/op 1.14
BLS aggregatePubkeys 32 - blst-native 39.666 us/op 34.781 us/op 1.14
BLS aggregatePubkeys 128 - blst-native 154.02 us/op 135.12 us/op 1.14
getAttestationsForBlock 88.546 ms/op 80.982 ms/op 1.09
CheckpointStateCache - add get delete 20.482 us/op 16.839 us/op 1.22
validate gossip signedAggregateAndProof - struct 4.6247 ms/op 4.5069 ms/op 1.03
validate gossip signedAggregateAndProof - treeBacked 6.6055 ms/op 4.4102 ms/op 1.50
validate gossip attestation - struct 3.6926 ms/op 2.0885 ms/op 1.77
validate gossip attestation - treeBacked 2.3369 ms/op 2.1195 ms/op 1.10
bytes32 toHexString 1.7290 us/op 1.6900 us/op 1.02
bytes32 Buffer.toString(hex) 704.00 ns/op 719.00 ns/op 0.98
bytes32 Buffer.toString(hex) from Uint8Array 908.00 ns/op 892.00 ns/op 1.02
bytes32 Buffer.toString(hex) + 0x 697.00 ns/op 719.00 ns/op 0.97
Object access 1 prop 0.34100 ns/op 0.33500 ns/op 1.02
Map access 1 prop 0.29600 ns/op 0.28800 ns/op 1.03
Object get x1000 17.543 ns/op 17.846 ns/op 0.98
Map get x1000 0.97600 ns/op 1.0020 ns/op 0.97
Object set x1000 109.19 ns/op 110.43 ns/op 0.99
Map set x1000 64.040 ns/op 64.005 ns/op 1.00
Return object 10000 times 0.37650 ns/op 0.37260 ns/op 1.01
Throw Error 10000 times 5.9368 us/op 5.9777 us/op 0.99
enrSubnets - fastDeserialize 64 bits 1.2390 us/op 1.2400 us/op 1.00
enrSubnets - ssz BitVector 64 bits 16.785 us/op 16.761 us/op 1.00
enrSubnets - fastDeserialize 4 bits 445.00 ns/op 452.00 ns/op 0.98
enrSubnets - ssz BitVector 4 bits 3.0010 us/op 2.9020 us/op 1.03
RateTracker 1000000 limit, 1 obj count per request 185.26 ns/op 178.83 ns/op 1.04
RateTracker 1000000 limit, 2 obj count per request 134.66 ns/op 132.12 ns/op 1.02
RateTracker 1000000 limit, 4 obj count per request 112.31 ns/op 110.96 ns/op 1.01
RateTracker 1000000 limit, 8 obj count per request 101.27 ns/op 98.311 ns/op 1.03
RateTracker with prune 4.2290 us/op 3.8480 us/op 1.10

by benchmarkbot/action

@@ -88,8 +88,8 @@ function resolveKeystorePaths(fileOrDirPath: string): string[] {
if (fs.lstatSync(fileOrDirPath).isDirectory()) {
return fs
.readdirSync(fileOrDirPath)
.map((file) => path.join(fileOrDirPath, file))
.filter((filepath) => filepath.endsWith(".json"));
.filter((file) => file.indexOf("deposit_data") === -1 && file.endsWith(".json"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Regex expression is typically much faster than indexOf and endsWith and much more idiomatic

Copy link
Contributor Author

@dadepo dadepo Jan 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Regex expression is typically much faster than indexOf and endsWith

I think that is debatable. The benchmarks I see (not ones I run myself) suggests this is not the case.

and much more idiomatic

Using regex is definitely more succinct, and since this is not a hot code path, I guess it is fine to go with the regex.

I have updated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point I haven't benchmarked it 😆

@dapplion
Copy link
Contributor

Can you add a couple unit tests for resolveKeystorePaths()?

@dadepo dadepo requested a review from dapplion January 18, 2022 12:19
Copy link
Contributor

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! ❤️

@dapplion dapplion merged commit 355ef6d into master Jan 18, 2022
@dapplion dapplion deleted the dadepo/prevent-app-crash-when-file-not-keystore branch January 18, 2022 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Validator handler is expecting objects only in /keystores
2 participants