Skip to content

Commit

Permalink
Document SSZ testing, and test-suite running
Browse files Browse the repository at this point in the history
  • Loading branch information
protolambda committed Apr 18, 2019
1 parent 0f87f35 commit d7d2eea
Show file tree
Hide file tree
Showing 6 changed files with 72 additions and 15 deletions.
21 changes: 21 additions & 0 deletions specs/test_formats/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,3 +175,24 @@ To prevent parsing of hundreds of different YAML files to test a specific test t
│   ... <--- more handlers
... <--- more test types
```


## Note for implementers

The basic pattern for test-suite loading and running is:

Iterate suites for given test-type, or sub-type (e.g. `operations > deposits`):
1. Filter test-suite, options:
- Config: Load first few lines, load into YAML, and check `config`, either:
- Pass the suite to the correct compiled target
- Ignore the suite if running tests as part of a compiled target with different configuration
- Load the correct configuration for the suite dynamically before running the suite
- Select by file name
- Filter for specific suites (e.g. for a specific fork)
2. Load the YAML
- Optionally translate the data into applicable naming, e.g. `snake_case` to `PascalCase`
3. Iterate through the `test_cases`
4. Ask test-runner to allocate a new test-case (i.e. objectify the test-case, generalize it with a `TestCase` interface)
Optionally pass raw test-case data to enable dynamic test-case allocation.
1. Load test-case data into it.
2. Make the test-case run.
15 changes: 0 additions & 15 deletions specs/test_formats/ssz/README.md

This file was deleted.

20 changes: 20 additions & 0 deletions specs/test_formats/ssz_generic/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# SSZ, generic tests

This set of test-suites provides general testing for SSZ:
to instantiate any container/list/vector/other type from binary data.

Since SSZ is in a development-phase, not the full suite of features is covered yet.
Note that these tests are based on the older SSZ package.
The tests are still relevant, but limited in scope:
more complex object encodings have changed since the original SSZ testing.

A minimal but useful series of tests covering `uint` encoding and decoding is provided.
This is a direct port of the older SSZ `uint` tests (minus outdated test cases).

[uint test format](./uint.md).

Note: the current phase-0 spec does not use larger uints, and uses byte vectors (fixed length) instead to represent roots etc.
The exact uint lengths to support may be redefined in the future.

Extension of the SSZ tests collection is planned, with an update to the new spec-maintained `minimal_ssz.py`,
see CI/testing issues for progress tracking.
File renamed without changes.
8 changes: 8 additions & 0 deletions specs/test_formats/ssz_static/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# SSZ, static tests

This set of test-suites provides static testing for SSZ:
to instantiate just the known ETH-2.0 SSZ types from binary data.

This series of tests is based on the spec-maintained `minimal_ssz.py`, i.e. fully consistent with the SSZ spec.

Test format documentation can be found here: [core test format](./core.md).
23 changes: 23 additions & 0 deletions specs/test_formats/ssz_static/core.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Test format: SSZ static types

The goal of this type is to provide clients with a solid reference how the known SSZ objects should be encoded.
Each object described in the Phase-0 spec is covered.
This is important, as many of the clients aiming to serialize/deserialize objects directly into structs/classes
do not support (or have alternatives for) generic SSZ encoding/decoding.
This test-format ensures these direct serializations are covered.

## Test case format

```yaml
type_name: string -- string, object name, formatted as in spec. E.g. "BeaconBlock"
value: dynamic -- the YAML-encoded value, of the type specified by type_name.
serialized: bytes -- string, SSZ-serialized data, hex encoded, with prefix 0x
root: bytes32 -- string, hash-tree-root of the value, hex encoded, with prefix 0x
```

## Condition

A test-runner can implement the following assertions:
- Serialization: After parsing the `value`, SSZ-serialize it: the output should match `serialized`
- Hash-tree-root: After parsing the `value`, Hash-tree-root it: the output should match `root`
- Deserialization: SSZ-deserialize the `serialized` value, and see if it matches the parsed `value`

0 comments on commit d7d2eea

Please sign in to comment.