Add new decorators to skip tests and handle it in testgen #2016
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sorry, 3 different parts in this PR.
Issue 1 (minor)
get_custody_secret
is duplicate.Solution for Issue 1
Use v-guide's
get_custody_secret
in test helper.Issue 2
For some test cases, it can't be passed with mainnet configs.
Solution for Issue 2
with_configs
decorator to assign available configsonly_full_crosslink
decorator to detect if the configuration cando full crosslinking: some test cases assume that each slot has enough committee to form crosslink.
CONFIG_NAME
to the config files."mainnet"
or"minimal"
.Issue 3
pytest
.pytest
to run the tests.It would be better to handle the "unavailable" forks and configs properly. e.g., some test cases are only for phase 0, or some test cases are only for mainnet configs.
Solution for Issue 3
Add
context.is_pytest
flag: True if calling viapytest
. False if calling from test generator.Add
dump_skipping_message
to dump message topytest
caller or test generator runner:pytest
: mark it isskip
. For example, if we callpytest --disable-bls --config=mainnet eth2spec/test/phase1/sanity/test_blocks.py
, it will show some test cases are skipped:SkippedTest
exception will be caught and show message like:Add
pytest
togen_helpers
requirements. But note that it's just becausepytest
is imported ineth2spec/test/context.py
, the test generator does NOT use it.