Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move spec tests to main lodestar package #3544

Merged
merged 6 commits into from
Dec 23, 2021
Merged

Move spec tests to main lodestar package #3544

merged 6 commits into from
Dec 23, 2021

Conversation

dapplion
Copy link
Contributor

Motivation

The Lodestar monorepo currently has a shadow package "spec-test-runner" that only exists to run spec tests.

Spec tests must be hosted somewhere in the monorepo and be run in CI. They could be in any package, since dependency relations don't affect the test folder.

Description

  • Move spec tests to the lodestar package since that's the "root" or "main" package of the monorepo.
  • Remove shadow package 'spec-test-runner'

@codeclimate
Copy link

codeclimate bot commented Dec 21, 2021 •
edited

Code Climate has analyzed commit 5bb8ebb and detected 0 issues on this pull request.

View more on Code Climate.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 21, 2021 •
edited

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: ec99ed9 Previous: 148ac6f Ratio
BeaconState.hashTreeRoot - No change 536.00 ns/op 545.00 ns/op 0.98
BeaconState.hashTreeRoot - 1 full validator 127.97 us/op 123.03 us/op 1.04
BeaconState.hashTreeRoot - 32 full validator 1.9174 ms/op 1.9424 ms/op 0.99
BeaconState.hashTreeRoot - 512 full validator 25.114 ms/op 26.262 ms/op 0.96
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 122.59 us/op 125.66 us/op 0.98
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.0862 ms/op 2.0622 ms/op 1.01
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 27.888 ms/op 29.085 ms/op 0.96
BeaconState.hashTreeRoot - 1 balances 92.579 us/op 93.683 us/op 0.99
BeaconState.hashTreeRoot - 32 balances 770.60 us/op 731.44 us/op 1.05
BeaconState.hashTreeRoot - 512 balances 7.3329 ms/op 7.7254 ms/op 0.95
BeaconState.hashTreeRoot - 250000 balances 135.73 ms/op 137.36 ms/op 0.99
processSlot - 1 slots 48.627 us/op 49.786 us/op 0.98
processSlot - 32 slots 2.9610 ms/op 2.9474 ms/op 1.00
getCommitteeAssignments - req 1 vs - 250000 vc 5.2929 ms/op 5.2752 ms/op 1.00
getCommitteeAssignments - req 100 vs - 250000 vc 7.3946 ms/op 7.4177 ms/op 1.00
getCommitteeAssignments - req 1000 vs - 250000 vc 7.9145 ms/op 7.8840 ms/op 1.00
computeProposers - vc 250000 22.055 ms/op 21.184 ms/op 1.04
computeEpochShuffling - vc 250000 187.24 ms/op 207.23 ms/op 0.90
getNextSyncCommittee - vc 250000 347.19 ms/op 344.53 ms/op 1.01
altair processAttestation - 250000 vs - 7PWei normalcase 43.731 ms/op 44.501 ms/op 0.98
altair processAttestation - 250000 vs - 7PWei worstcase 45.295 ms/op 47.270 ms/op 0.96
altair processAttestation - setStatus - 1/6 committees join 10.355 ms/op 14.894 ms/op 0.70
altair processAttestation - setStatus - 1/3 committees join 22.165 ms/op 26.148 ms/op 0.85
altair processAttestation - setStatus - 1/2 committees join 35.126 ms/op 38.234 ms/op 0.92
altair processAttestation - setStatus - 2/3 committees join 46.432 ms/op 53.181 ms/op 0.87
altair processAttestation - setStatus - 4/5 committees join 57.232 ms/op 69.956 ms/op 0.82
altair processAttestation - setStatus - 100% committees join 73.822 ms/op 78.365 ms/op 0.94
altair processAttestation - updateEpochParticipants - 1/6 committees join 12.050 ms/op 13.261 ms/op 0.91
altair processAttestation - updateEpochParticipants - 1/3 committees join 27.374 ms/op 27.694 ms/op 0.99
altair processAttestation - updateEpochParticipants - 1/2 committees join 26.359 ms/op 25.636 ms/op 1.03
altair processAttestation - updateEpochParticipants - 2/3 committees join 28.050 ms/op 26.161 ms/op 1.07
altair processAttestation - updateEpochParticipants - 4/5 committees join 34.722 ms/op 29.189 ms/op 1.19
altair processAttestation - updateEpochParticipants - 100% committees join 31.128 ms/op 30.221 ms/op 1.03
altair processAttestation - updateAllStatus 22.550 ms/op 26.903 ms/op 0.84
altair processBlock - 250000 vs - 7PWei normalcase 46.873 ms/op 46.093 ms/op 1.02
altair processBlock - 250000 vs - 7PWei worstcase 142.03 ms/op 119.91 ms/op 1.18
altair processEpoch - mainnet_e81889 1.0717 s/op 1.0922 s/op 0.98
mainnet_e81889 - altair beforeProcessEpoch 363.00 ms/op 368.43 ms/op 0.99
mainnet_e81889 - altair processJustificationAndFinalization 126.08 us/op 578.87 us/op 0.22
mainnet_e81889 - altair processInactivityUpdates 19.219 ms/op 18.715 ms/op 1.03
mainnet_e81889 - altair processRewardsAndPenalties 148.55 ms/op 147.47 ms/op 1.01
mainnet_e81889 - altair processRegistryUpdates 7.6970 us/op 5.4920 us/op 1.40
mainnet_e81889 - altair processSlashings 2.1770 us/op 1.1590 us/op 1.88
mainnet_e81889 - altair processEth1DataReset 2.0300 us/op 1.0860 us/op 1.87
mainnet_e81889 - altair processEffectiveBalanceUpdates 10.915 ms/op 11.814 ms/op 0.92
mainnet_e81889 - altair processSlashingsReset 18.168 us/op 9.0570 us/op 2.01
mainnet_e81889 - altair processRandaoMixesReset 23.396 us/op 11.890 us/op 1.97
mainnet_e81889 - altair processHistoricalRootsUpdate 2.8880 us/op 1.3230 us/op 2.18
mainnet_e81889 - altair processParticipationFlagUpdates 103.91 ms/op 105.72 ms/op 0.98
mainnet_e81889 - altair processSyncCommitteeUpdates 2.2290 us/op 1.0560 us/op 2.11
mainnet_e81889 - altair afterProcessEpoch 225.07 ms/op 226.17 ms/op 1.00
altair processInactivityUpdates - 250000 normalcase 74.094 ms/op 75.834 ms/op 0.98
altair processInactivityUpdates - 250000 worstcase 69.097 ms/op 68.783 ms/op 1.00
altair processParticipationFlagUpdates - 250000 anycase 103.89 ms/op 105.16 ms/op 0.99
altair processRewardsAndPenalties - 250000 normalcase 156.51 ms/op 152.38 ms/op 1.03
altair processRewardsAndPenalties - 250000 worstcase 151.15 ms/op 148.83 ms/op 1.02
altair processSyncCommitteeUpdates - 250000 372.78 ms/op 356.31 ms/op 1.05
Tree 40 250000 create 859.98 ms/op 860.19 ms/op 1.00
Tree 40 250000 get(125000) 321.36 ns/op 328.88 ns/op 0.98
Tree 40 250000 set(125000) 1.8278 us/op 1.9807 us/op 0.92
Tree 40 250000 toArray() 36.845 ms/op 46.541 ms/op 0.79
Tree 40 250000 iterate all - toArray() + loop 44.780 ms/op 39.178 ms/op 1.14
Tree 40 250000 iterate all - get(i) 118.14 ms/op 120.17 ms/op 0.98
MutableVector 250000 create 24.153 ms/op 21.353 ms/op 1.13
MutableVector 250000 get(125000) 13.608 ns/op 15.332 ns/op 0.89
MutableVector 250000 set(125000) 541.85 ns/op 580.61 ns/op 0.93
MutableVector 250000 toArray() 8.3312 ms/op 8.6625 ms/op 0.96
MutableVector 250000 iterate all - toArray() + loop 8.4560 ms/op 8.6272 ms/op 0.98
MutableVector 250000 iterate all - get(i) 3.4293 ms/op 3.4329 ms/op 1.00
Array 250000 create 5.1117 ms/op 5.4464 ms/op 0.94
Array 250000 clone - spread 2.4388 ms/op 1.7741 ms/op 1.37
Array 250000 get(125000) 1.0810 ns/op 0.89500 ns/op 1.21
Array 250000 set(125000) 1.0890 ns/op 0.88800 ns/op 1.23
Array 250000 iterate all - loop 167.82 us/op 167.83 us/op 1.00
aggregationBits - 2048 els - readonlyValues 241.03 us/op 206.50 us/op 1.17
aggregationBits - 2048 els - zipIndexesInBitList 50.921 us/op 37.987 us/op 1.34
regular array get 100000 times 67.368 us/op 67.405 us/op 1.00
wrappedArray get 100000 times 67.425 us/op 67.455 us/op 1.00
arrayWithProxy get 100000 times 28.912 ms/op 27.567 ms/op 1.05
ssz.Root.equals 1.0980 us/op 1.1860 us/op 0.93
ssz.Root.equals with valueOf() 1.4050 us/op 1.2830 us/op 1.10
byteArrayEquals with valueOf() 1.3170 us/op 1.3420 us/op 0.98
phase0 processBlock - 250000 vs - 7PWei normalcase 10.783 ms/op 10.430 ms/op 1.03
phase0 processBlock - 250000 vs - 7PWei worstcase 79.114 ms/op 76.514 ms/op 1.03
phase0 afterProcessEpoch - 250000 vs - 7PWei 208.86 ms/op 203.72 ms/op 1.03
phase0 beforeProcessEpoch - 250000 vs - 7PWei 617.04 ms/op 554.63 ms/op 1.11
phase0 processEpoch - mainnet_e58758 807.60 ms/op 797.07 ms/op 1.01
mainnet_e58758 - phase0 beforeProcessEpoch 494.02 ms/op 509.93 ms/op 0.97
mainnet_e58758 - phase0 processJustificationAndFinalization 77.628 us/op 49.975 us/op 1.55
mainnet_e58758 - phase0 processRewardsAndPenalties 174.63 ms/op 103.53 ms/op 1.69
mainnet_e58758 - phase0 processRegistryUpdates 65.579 us/op 48.100 us/op 1.36
mainnet_e58758 - phase0 processSlashings 2.6700 us/op 1.7540 us/op 1.52
mainnet_e58758 - phase0 processEth1DataReset 2.3080 us/op 1.0240 us/op 2.25
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 9.1953 ms/op 9.4637 ms/op 0.97
mainnet_e58758 - phase0 processSlashingsReset 14.967 us/op 10.020 us/op 1.49
mainnet_e58758 - phase0 processRandaoMixesReset 17.936 us/op 14.099 us/op 1.27
mainnet_e58758 - phase0 processHistoricalRootsUpdate 2.0130 us/op 1.1330 us/op 1.78
mainnet_e58758 - phase0 processParticipationRecordUpdates 14.540 us/op 14.210 us/op 1.02
mainnet_e58758 - phase0 afterProcessEpoch 180.37 ms/op 183.03 ms/op 0.99
phase0 processEffectiveBalanceUpdates - 250000 normalcase 11.169 ms/op 10.771 ms/op 1.04
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.3584 s/op 1.4481 s/op 0.94
phase0 processRegistryUpdates - 250000 normalcase 60.440 us/op 39.311 us/op 1.54
phase0 processRegistryUpdates - 250000 badcase_full_deposits 3.0141 ms/op 3.1470 ms/op 0.96
phase0 processRegistryUpdates - 250000 worstcase 0.5 1.7468 s/op 1.7379 s/op 1.01
phase0 getAttestationDeltas - 250000 normalcase 35.753 ms/op 35.863 ms/op 1.00
phase0 getAttestationDeltas - 250000 worstcase 35.337 ms/op 39.801 ms/op 0.89
phase0 processSlashings - 250000 worstcase 35.301 ms/op 38.697 ms/op 0.91
shuffle list - 16384 els 12.638 ms/op 12.676 ms/op 1.00
shuffle list - 250000 els 182.20 ms/op 183.83 ms/op 0.99
getEffectiveBalances - 250000 vs - 7PWei 10.710 ms/op 10.441 ms/op 1.03
computeDeltas 3.9421 ms/op 4.0696 ms/op 0.97
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.2662 ms/op 2.5524 ms/op 0.89
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 676.16 us/op 728.56 us/op 0.93
BLS verify - blst-native 1.8586 ms/op 1.8593 ms/op 1.00
BLS verifyMultipleSignatures 3 - blst-native 3.8149 ms/op 3.8183 ms/op 1.00
BLS verifyMultipleSignatures 8 - blst-native 8.2275 ms/op 8.2317 ms/op 1.00
BLS verifyMultipleSignatures 32 - blst-native 29.873 ms/op 29.899 ms/op 1.00
BLS aggregatePubkeys 32 - blst-native 39.251 us/op 39.266 us/op 1.00
BLS aggregatePubkeys 128 - blst-native 153.63 us/op 153.82 us/op 1.00
getAttestationsForBlock 80.827 ms/op 81.586 ms/op 0.99
CheckpointStateCache - add get delete 15.347 us/op 15.998 us/op 0.96
validate gossip signedAggregateAndProof - struct 4.6047 ms/op 4.5122 ms/op 1.02
validate gossip signedAggregateAndProof - treeBacked 4.5420 ms/op 4.4084 ms/op 1.03
validate gossip attestation - struct 2.0936 ms/op 2.0855 ms/op 1.00
validate gossip attestation - treeBacked 2.1156 ms/op 2.1192 ms/op 1.00
Object access 1 prop 0.31600 ns/op 0.34500 ns/op 0.92
Map access 1 prop 0.27200 ns/op 0.27600 ns/op 0.99
Object get x1000 17.627 ns/op 17.289 ns/op 1.02
Map get x1000 0.99600 ns/op 0.97700 ns/op 1.02
Object set x1000 114.01 ns/op 118.94 ns/op 0.96
Map set x1000 69.489 ns/op 73.440 ns/op 0.95
Return object 10000 times 0.37590 ns/op 0.37420 ns/op 1.00
Throw Error 10000 times 6.0456 us/op 5.8917 us/op 1.03
RateTracker 1000000 limit, 1 obj count per request 185.84 ns/op 191.12 ns/op 0.97
RateTracker 1000000 limit, 2 obj count per request 140.13 ns/op 142.62 ns/op 0.98
RateTracker 1000000 limit, 4 obj count per request 111.42 ns/op 120.39 ns/op 0.93
RateTracker 1000000 limit, 8 obj count per request 100.38 ns/op 102.80 ns/op 0.98
RateTracker with prune 4.0090 us/op 4.6020 us/op 0.87

by benchmarkbot/action

@codecov
Copy link

codecov bot commented Dec 21, 2021 •
edited

Codecov Report

Merging #3544 (ec99ed9) into master (148ac6f) will not change coverage.
The diff coverage is n/a.

❗ Current head ec99ed9 differs from pull request most recent head 5bb8ebb. Consider uploading reports for the commit 5bb8ebb to get more accurate results

@@           Coverage Diff           @@
##           master    #3544   +/-   ##
=======================================
  Coverage   37.56%   37.56%           
=======================================
  Files         310      310           
  Lines        8253     8253           
  Branches     1273     1273           
=======================================
  Hits         3100     3100           
  Misses       5005     5005           
  Partials      148      148

@dapplion dapplion merged commit 97188a4 into master Dec 23, 2021
@dapplion dapplion deleted the dapplion/move-spec-tests branch December 23, 2021 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@twoeths twoeths twoeths approved these changes

@g11tech g11tech g11tech approved these changes

@mpetrunic mpetrunic Awaiting requested review from mpetrunic

@wemeetagain wemeetagain Awaiting requested review from wemeetagain

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@dapplion @twoeths @g11tech