Merge pull request #1114 from ethereum/finish-state-tests-intro

Finish state tests intro

specs/test_formats/epoch_processing/README.md

@@ -0,0 +1,34 @@
+# Epoch processing tests
+
+The different epoch sub-transitions are tested individually with test handlers.
+The format is similar to block-processing state-transition tests.
+There is no "change" factor however, the transitions are pure functions with just the pre-state as input.
+Hence, the format is shared between each test-handler. (See test condition documentation on how to run the tests.)
+
+## Test case format
+
+```yaml
+description: string    -- description of test case, purely for debugging purposes
+bls_required: bool     -- optional, true if the test validity is strictly dependent on BLS being ON. False otherwise.
+bls_ignored: bool      -- optional, true if the test validity is strictly dependent on BLS being OFF. False otherwise.
+pre: BeaconState       -- state before running the sub-transition
+post: BeaconState      -- state after applying the epoch sub-transition.
+```
+
+Note: if both `bls_required` and `bls_ignored` are false (or simply not included),
+ then the test consumer can freely choose to run with BLS ON or OFF.
+One may choose for OFF for performance reasons during repeated testing. Otherwise it is recommended to run with BLS ON.
+
+## Condition
+
+A handler of the `epoch_processing` test-runner should process these cases, 
+ calling the corresponding processing implementation.
+
+Sub-transitions:
+
+| *`sub-transition-name`* | *`processing call`*               |
+|-------------------------|-----------------------------------|
+| `crosslinks`            | `process_crosslinks(state)`       |
+| `registry_updates`      | `process_registry_updates(state)` |
+
+The resulting state should match the expected `post` state.

specs/test_formats/operations/README.md

@@ -2,9 +2,39 @@
 
 The different kinds of operations ("transactions") are tested individually with test handlers.
 
-The tested operation kinds are:
-- [`deposits`](./deposits.md)
-- More tests are work-in-progress.
+## Test case format
 
+```yaml
+description: string                    -- description of test case, purely for debugging purposes
+bls_required: bool                     -- optional, true if the test validity is strictly dependent on BLS being ON. False otherwise.
+bls_ignored: bool                      -- optional, true if the test validity is strictly dependent on BLS being OFF. False otherwise.
+pre: BeaconState                       -- state before applying the operation
+<operation-name>: <operation-object>   -- the YAML encoded operation, e.g. a "ProposerSlashing", or "Deposit".
+post: BeaconState                      -- state after applying the operation. No value if operation processing is aborted.
+```
 
+Note: if both `bls_required` and `bls_ignored` are false (or simply not included),
+ then the test consumer can freely choose to run with BLS ON or OFF.
+One may choose for OFF for performance reasons during repeated testing. Otherwise it is recommended to run with BLS ON.
 
+## Condition
+
+A handler of the `operations` test-runner should process these cases, 
+ calling the corresponding processing implementation.
+
+Operations:
+
+| *`operation-name`*      | *`operation-object`* | *`input name`*       | *`processing call`*                                    |
+|-------------------------|----------------------|----------------------|--------------------------------------------------------|
+| `attestation`           | `Attestation`        | `attestation`        | `process_attestation(state, attestation)`                  |
+| `attester_slashing`     | `AttesterSlashing`   | `attester_slashing`  | `process_attester_slashing(state, attester_slashing)`            |
+| `block_header`          | `Block`              | `block`              | `process_block_header(state, block)`                   |
+| `deposit`               | `Deposit`            | `deposit`            | `process_deposit(state, deposit)`                      |
+| `proposer_slashing`     | `ProposerSlashing`   | `proposer_slashing`  | `process_proposer_slashing(state, proposer_slashing)`  |
+| `transfer`              | `Transfer`           | `transfer`           | `process_transfer(state, transfer)`                    |
+| `voluntary_exit`        | `VoluntaryExit`      | `voluntary_exit`     | `process_voluntary_exit(state, voluntary_exit)`        |
+
+Note that `block_header` is not strictly an operation (and is a full `Block`), but processed in the same manner, and hence included here. 
+
+The resulting state should match the expected `post` state, or if the `post` state is left blank,
+ the handler should reject the input operation as invalid.

specs/test_formats/sanity/README.md

@@ -0,0 +1,7 @@
+# Sanity tests
+
+The aim of the sanity tests is to set a base-line on what really needs to pass, i.e. the essentials.
+
+There are two handlers, documented individually:
+- [`slots`](./slots.md): transitions of one or more slots (and epoch transitions within)
+- [`blocks`](./blocks.md): transitions triggered by one or more blocks

specs/test_formats/sanity/blocks.md

@@ -0,0 +1,19 @@
+# Sanity blocks testing
+
+Sanity tests to cover a series of one or more blocks being processed, aiming to cover common changes.
+
+## Test case format
+
+```yaml
+description: string    -- description of test case, purely for debugging purposes
+bls_required: bool     -- optional, true if the test validity is strictly dependent on BLS being ON. False otherwise.
+bls_ignored: bool      -- optional, true if the test validity is strictly dependent on BLS being OFF. False otherwise.
+pre: BeaconState       -- state before running through the transitions triggered by the blocks.
+blocks: [BeaconBlock]  -- blocks to process, in given order, following the main transition function (i.e. process slot and epoch transitions in between blocks as normal)
+post: BeaconState      -- state after applying all the transitions triggered by the blocks.
+```
+
+## Condition
+
+The resulting state should match the expected `post` state, or if the `post` state is left blank,
+ the handler should reject the series of blocks as invalid.

specs/test_formats/sanity/slots.md

@@ -0,0 +1,24 @@
+# Sanity slots testing
+
+Sanity tests to cover a series of one or more empty-slot transitions being processed, aiming to cover common changes.
+
+## Test case format
+
+```yaml
+description: string    -- description of test case, purely for debugging purposes
+bls_required: bool     -- optional, true if the test validity is strictly dependent on BLS being ON. False otherwise.
+bls_ignored: bool      -- optional, true if the test validity is strictly dependent on BLS being OFF. False otherwise.
+pre: BeaconState       -- state before running through the transitions.
+slots: N               -- amount of slots to process, N being a positive numer.
+post: BeaconState      -- state after applying all the transitions.
+```
+
+The transition with pure time, no blocks, is known as `state_transition_to(state, slot)` in the spec.
+This runs state-caching (pure slot transition) and epoch processing (every E slots).
+
+To process the data, call `state_transition_to(pre, pre.slot + N)`. And see if `pre` mutated into the equivalent of `post`.
+
+
+## Condition
+
+The resulting state should match the expected `post` state.

test_generators/epoch_processing/README.md

@@ -0,0 +1,11 @@
+# Epoch processing
+
+Epoch processing covers the sub-transitions during an epoch change.
+
+An epoch-processing test-runner can consume these sub-transition test-suites,
+ and handle different kinds of epoch sub-transitions by processing the cases using the specified test handler.
+
+Information on the format of the tests can be found in the [epoch-processing test formats documentation](../../specs/test_formats/epoch_processing/README.md).
+
+
+

test_generators/epoch_processing/main.py

@@ -0,0 +1,38 @@
+from typing import Callable, Iterable
+
+from eth2spec.phase0 import spec
+from eth2spec.test.epoch_processing import (
+    test_process_crosslinks,
+    test_process_registry_updates
+)
+from gen_base import gen_runner, gen_suite, gen_typing
+from gen_from_tests.gen import generate_from_tests
+from preset_loader import loader
+
+
+def create_suite(transition_name: str, config_name: str, get_cases: Callable[[], Iterable[gen_typing.TestCase]]) \
+        -> Callable[[str], gen_typing.TestSuiteOutput]:
+    def suite_definition(configs_path: str) -> gen_typing.TestSuiteOutput:
+        presets = loader.load_presets(configs_path, config_name)
+        spec.apply_constants_preset(presets)
+
+        return ("%s_%s" % (transition_name, config_name), transition_name, gen_suite.render_suite(
+            title="%s epoch processing" % transition_name,
+            summary="Test suite for %s type epoch processing" % transition_name,
+            forks_timeline="testing",
+            forks=["phase0"],
+            config=config_name,
+            runner="epoch_processing",
+            handler=transition_name,
+            test_cases=get_cases()))
+
+    return suite_definition
+
+
+if __name__ == "__main__":
+    gen_runner.run_generator("epoch_processing", [
+        create_suite('crosslinks', 'minimal', lambda: generate_from_tests(test_process_crosslinks)),
+        create_suite('crosslinks', 'mainnet', lambda: generate_from_tests(test_process_crosslinks)),
+        create_suite('registry_updates', 'minimal', lambda: generate_from_tests(test_process_registry_updates)),
+        create_suite('registry_updates', 'mainnet', lambda: generate_from_tests(test_process_registry_updates)),
+    ])

test_generators/epoch_processing/requirements.txt

@@ -0,0 +1,4 @@
+eth-utils==1.6.0
+../../test_libs/gen_helpers
+../../test_libs/config_helpers
+../../test_libs/pyspec

test_generators/operations/README.md

@@ -3,7 +3,6 @@
 Operations (or "transactions" in previous spec iterations),
  are atomic changes to the state, introduced by embedding in blocks.
 
-This generator provides a series of test suites, divided into handler, for each operation type.
 An operation test-runner can consume these operation test-suites,
  and handle different kinds of operations by processing the cases using the specified test handler.
 

test_generators/operations/main.py

@@ -1,3 +1,5 @@
+from typing import Callable, Iterable
+
 from eth2spec.test.block_processing import (
     test_process_attestation,
     test_process_attester_slashing,
@@ -8,9 +10,29 @@
     test_process_voluntary_exit
 )
 
-from gen_base import gen_runner
+from gen_base import gen_runner, gen_suite, gen_typing
+from gen_from_tests.gen import generate_from_tests
+from preset_loader import loader
+from eth2spec.phase0 import spec
+
+
+def create_suite(operation_name: str, config_name: str, get_cases: Callable[[], Iterable[gen_typing.TestCase]]) \
+        -> Callable[[str], gen_typing.TestSuiteOutput]:
+    def suite_definition(configs_path: str) -> gen_typing.TestSuiteOutput:
+        presets = loader.load_presets(configs_path, config_name)
+        spec.apply_constants_preset(presets)
+
+        return ("%s_%s" % (operation_name, config_name), operation_name, gen_suite.render_suite(
+            title="%s operation" % operation_name,
+            summary="Test suite for %s type operation processing" % operation_name,
+            forks_timeline="testing",
+            forks=["phase0"],
+            config=config_name,
+            runner="operations",
+            handler=operation_name,
+            test_cases=get_cases()))
+    return suite_definition
 
-from suite_creator import generate_from_tests, create_suite
 
 if __name__ == "__main__":
     gen_runner.run_generator("operations", [

test_generators/sanity/README.md

@@ -0,0 +1,8 @@
+# Sanity tests
+
+Sanity tests cover regular state-transitions in a common block-list format, to ensure the basics work.
+
+Information on the format of the tests can be found in the [sanity test formats documentation](../../specs/test_formats/sanity/README.md).
+
+
+

test_generators/sanity/main.py

@@ -0,0 +1,35 @@
+from typing import Callable, Iterable
+
+from eth2spec.test.sanity import test_blocks, test_slots
+
+from gen_base import gen_runner, gen_suite, gen_typing
+from gen_from_tests.gen import generate_from_tests
+from preset_loader import loader
+from eth2spec.phase0 import spec
+
+
+def create_suite(handler_name: str, config_name: str, get_cases: Callable[[], Iterable[gen_typing.TestCase]]) \
+        -> Callable[[str], gen_typing.TestSuiteOutput]:
+    def suite_definition(configs_path: str) -> gen_typing.TestSuiteOutput:
+        presets = loader.load_presets(configs_path, config_name)
+        spec.apply_constants_preset(presets)
+
+        return ("%sanity_s_%s" % (handler_name, config_name), handler_name, gen_suite.render_suite(
+            title="sanity testing",
+            summary="Sanity test suite, %s type, generated from pytests" % handler_name,
+            forks_timeline="testing",
+            forks=["phase0"],
+            config=config_name,
+            runner="sanity",
+            handler=handler_name,
+            test_cases=get_cases()))
+    return suite_definition
+
+
+if __name__ == "__main__":
+    gen_runner.run_generator("sanity", [
+        create_suite('blocks', 'minimal', lambda: generate_from_tests(test_blocks)),
+        create_suite('blocks', 'mainnet', lambda: generate_from_tests(test_blocks)),
+        create_suite('slots', 'minimal', lambda: generate_from_tests(test_slots)),
+        create_suite('slots', 'mainnet', lambda: generate_from_tests(test_slots)),
+    ])

test_generators/sanity/requirements.txt

@@ -0,0 +1,4 @@
+eth-utils==1.6.0
+../../test_libs/gen_helpers
+../../test_libs/config_helpers
+../../test_libs/pyspec

test_libs/gen_helpers/gen_from_tests/gen.py

@@ -0,0 +1,25 @@
+from inspect import getmembers, isfunction
+
+def generate_from_tests(src, bls_active=True):
+    """
+    Generate a list of test cases by running tests from the given src in generator-mode.
+    :param src: to retrieve tests from (discovered using inspect.getmembers)
+    :param bls_active: optional, to override BLS switch preference. Defaults to True.
+    :return: the list of test cases.
+    """
+    fn_names = [
+        name for (name, _) in getmembers(src, isfunction)
+        if name.startswith('test_')
+    ]
+    out = []
+    print("generating test vectors from tests source: %s" % src.__name__)
+    for name in fn_names:
+        tfn = getattr(src, name)
+        try:
+            test_case = tfn(generator_mode=True, bls_active=bls_active)
+            # If no test case data is returned, the test is ignored.
+            if test_case is not None:
+                out.append(test_case)
+        except AssertionError:
+            print("ERROR: failed to generate vector from test: %s (src: %s)" % (name, src.__name__))
+    return out

test_libs/gen_helpers/setup.py

@@ -2,7 +2,7 @@
 
 setup(
     name='gen_helpers',
-    packages=['gen_base'],
+    packages=['gen_base', 'gen_from_tests'],
     install_requires=[
         "ruamel.yaml==0.15.96",
         "eth-utils==1.6.0"