tests: adapt JSON suite for cross-client, cross-chain testing

[meowsbits]

Building an idea around adapting multi-geth in coordination with ethereum/tests to use (parts of) the existing JSON ethereum/tests suite to develop cross-client and multi-network available tests.

Hurdles:

• Current suite uses Fork-derived expectations, which is severely limiting to the extensibility of these tests beyond those precise chain configuration definitions.

Goals:

• A test suite comparable to 'coverage' in ethereum/tests, but extensible across chain configurations.
• These improved tests (and their schema patterns) should then be available for adoption between multi-geth and parity-ethereum at least.

These goals together would allow x-client/x-chain testing, which would be really nice to have for substantiating confidence around forthcoming hardforks, proving feature implementations against spec, etc.

• TODO: continue to flesh out this idea and documentation

[whilei]

/*

• Tests should run respective to chain configurations.
• Chain configurations should be integrated as test data.
• Chain configurations which are intergrated as test data should be spec'd,
  so that any client can use those configurations in their feature switch implementations
  and thus actually use the tests.

Operations:

• Define a chain configuration schema (JSON). This must be expected to play nicely with multi-geth and parity-ethereum.
  • AFAIR there might already exist a tool which can translate between configuration schemas;
    go-ethereum considers chain configuration as a facet of genesis block specification.
  • multi-geth already has JSON tags defined for params.ChainConfig
• Implement test suite that can run incorporated chain configurations defined under this schema
  against existing test chain configurations (aka Forks). If tests pass, we know that our test runner
  and chain configurations are playing well together to match existing chain configurations.
• Implement "test runner" that is actually a "test generator," which, instead of check got vs. want test results,
  will write test results to some file in JSON format.
• Using this test generator, create JSON tests against chain configurations beyond the currently hardcoded ETH-only
  magical snake charming tests. These should be able to prove that existing Parity and Go-ethereum (multi-geth)
  implementations pass tests equivalently for existing (live network) chain configurations.
• Implement a chain configuration generator that can create chain X test permutations, where chains include but
  are not limited to any network's canonical specifications. This will aid to ferret out edge cases.

*/

[meowsbits]

There are supposed schemas for JSON tests https://github.com/ethereum/tests/tree/develop/JSONSchema, but I haven't checked that they're actually implemented, and they are still relying on hardcoded opaque chain definitions.

• ABITests/ https://github.com/ethereum/tests/tree/develop/ABITests. Does not test w/ chain configs.
• BasicTests/ https://github.com/ethereum/tests/blob/develop/BasicTests/difficultyByzantium.json. Sometimes uses file names to denote associated chain config.
• BlockChainTests/ https://github.com/ethereum/tests/blob/develop/BlockchainTests/bcForgedTest/bcForkBlockTest.json#L35 uses Network JSON field to denote chain config per "subtest".
  • BlockchainTests/GeneralStateTests/ ditto.
  • https://github.com/ethereum/tests/blob/develop/BlockchainTests/TransitionTests/bcEIP158ToByzantium/ByzantiumTransition.json#L288 ditto.
• GeneralStateTests/ https://github.com/ethereum/tests/blob/develop/GeneralStateTests/stArgsZeroOneBalance/addNonConst.json#L18-L19 Uses object post.<chain config name> to denote corresponding outcomes for state changes.
• GenesisTests/ Does not test with chain configs.
• KeyStoreTests/ Can be skipped.
• PoWTests/ ditto.
• RLPTests/ ditto.
• Retesteth/ ditto.
• TransactionTests/ https://github.com/ethereum/tests/blob/develop/TransactionTests/ttData/DataTestEnoughGAS.json#L3 Use <Network> fields flatly adjacent to other fields, eg. _info.
• TrieTests/ Can be skipped.
• VMTests/ Do not denote chain configs. This might be b/c a certain implied default is assumed. Need more investigation.