[Blocked by upstream + pending SSZ test] Full EF state tests (if possible)

[mratsim]

PR to execute the full EF state tests in https://github.com/ethereum/eth2.0-tests/tree/33e762c76e0a9458d9d89fa4f9c696e769fb2e2f/state

Current status - Parsing the file into Nimbus datastructure works fine:

[Suite] Official - State tests
  [OK] Parsing the official state tests into Nimbus beacon types
  [OK] Initializing from scratch a new beacon chain with the same constants and deposit configuration as official state test 0
Deserialized state hash: 0x8436CCB3DFBF56400530E6C2C11C9B44CEB7BC0A4E0477B78E0B76C0470E9D7B
From-scratch state hash: 0x6B8ABD3E087D54495B74A4566AADC1BBC972B9470D4EA7CEEF1E04CA29E05357
  [OK] [For information] Comparing state hashes
Test #000: test_empty_block_transition
Test #001: test_skipped_slots
Test #002: test_empty_epoch_transition
Test #003: test_empty_epoch_transition_not_finalizing
Test #004: test_proposer_slashing
Test #005: test_attestation
Test #006: test_deposit_in_block
Test #007: test_deposit_top_up
Test #008: test_voluntary_exit
Test #009: test_transfer
Test #010: test_ejection
Test #011: test_historical_batch
  [OK] [For information]  Print list of official tests to implement

The reference transition functions are implemented here: https://github.com/ethereum/eth2.0-specs/blob/2baa242ac004b0475604c4c4ef4315e14f56c5c7/tests/phase0/test_sanity.py#L55-L460

[mratsim]

Unfortunately I can't make much progress on those tests at the moment:

1. The tests are using 0x00000000... signature everywhere which is an invalid compressed signature according to spec. A valid zero signature would be 0xc0000000...

2. In our codebase a deserializing check will prevent us from deserializing an invalid signature. We can work around that by using find/replace in the yaml/json files as done in repo https://github.com/status-im/nim-eth2-official-tests#commands. Raised Use valid compressed signatures in the state test ethereum/eth2.0-tests#27

3. However we have hash checks in state_transition that depends on BeaconBlockHeader that contains this signature, those checks fails if we change the signature by find/replace: https://github.com/status-im/nim-beacon-chain/blob/02daef60c6e99daba3a9679fe4958fb02b65c62a/beacon_chain/state_transition.nim#L38-L55

   https://github.com/status-im/nim-beacon-chain/blob/02daef60c6e99daba3a9679fe4958fb02b65c62a/beacon_chain/spec/datatypes.nim#L329-L335

4. Due to the previous 3 points, instead of testing the empty block transition, I wanted to test if at least we produced the previous_block_root hash from the test files when doing signed_root on a deserialized block header, but we do not, so we need SSZ tests. Pending Tree Hash Test Generator ethereum/eth2.0-test-generators#34 or the test generators rework Combine specs and test-generators ethereum/consensus-specs#851 for SSZ tests from the executable spec.

[arnetheduck]

FYI, both us and @paulhauner have raised 1 with @djrtwo a few times, hopefully that will change

SSZ is slated for a change: ethereum/consensus-specs#787

[mratsim]

SSZ test vectors pending merge: ethereum/consensus-specs#962

[Swader]

@mratsim ethereum/consensus-specs#962 has been merged