driver, runner and CI for interoperable test-vector based testing. #3081
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
raulk
force-pushed
the
conformance-tests
branch
from
40eefd4
to
ed33579
Compare
This commit introduces a new package `conformance` containing:
1. the test driver to exercise Lotus against interoperable
test vectors, and
2. the test runner, which integrates go test with the test vector
corpus hosted at https://github.com/filecoin-project/conformance-vectors.
The corpus is mounted via a git submodule.
Right now, only message-class test vectors are supported. In the
next week, this support will be extended to tipset-class, chain-class,
and block sequence-class vectors.
raulk
force-pushed
the
conformance-tests
branch
from
ed33579
to
322b331
Compare
raulk
force-pushed
the
conformance-tests
branch
from
9123bc0
to
58c340b
Compare
raulk
force-pushed
the
conformance-tests
branch
from
58c340b
to
f1c4527
Compare
raulk
changed the title
This was referenced Aug 18, 2020
This was referenced Aug 18, 2020
willscott
reviewed
Aug 19, 2020
This was referenced Aug 24, 2020
Closed
whyrusleeping
approved these changes
Aug 26, 2020
raulk
force-pushed
the
conformance-tests
branch
from
10bda4f
to
75e0387
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces a new package
conformancecontaining:corpuspackage.It also modifies the CircleCI configuration to add this as a separate suite, at least for now. This work is still stabilising, and some things may break. We don't want to add noise to the current suites.
The test vector corpus (https://github.com/filecoin-project/test-vectors/ repo) is mounted onto the Lotus repo via a git submodule.
Right now, only message-class test vectors are supported. In the next week, support will be extended to tipset-class, chain-class, and block sequence-class vectors.
Alerting on consensus-breaking changes
We are still pre-mainnet and we can afford to perform frequent network resets to introduce consensus-breaking changes (e.g. actor state data model or state manipulations). However, that luxury has an expiry date, and we need to level up our game in detecting breakages that can lead to forks.
Furthermore, other teams need to test their interoperability with us, in a gradual and targeted manner.
These test vectors are generated from the reference Filecoin implementation (Lotus), at a particular version. As such, they will alert us about regressions and consensus-breaking changes, such as: changes in state data model, changes to state transformations, changes in gas pricing, changes in exit codes, etc.
In some cases, the change might be intended; in others, it might not. That's the value of this: it will force a reflection/coordination point.
TODO