Berlin update fix1

The recent tests changes:

Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://github.com/ethereum/retesteth/wiki/Transition-Tool)
Previous release link: https://github.com/ethereum/tests/releases/tag/8.0.0

Test format

• Fillers
  tx.none : "auto" is fixed to work with 64bits in blockchain tests
  Nonce > 2**64 - 1 is not allowed now. same as block gasLimit

• Filled
  New field in state tests txbytes containing transaction rlp or rlp ( tx.type + tx.bytes)

Test Regeneration

Update all:
PR #805

• Migrate VMTests into StateTests format
  PR #802

• Fix the nonce bug in extcodehash test
  PR #804

• More tests around revert opcode
  PR #803

Upcoming changes:

• old VMTests and it's blockchain form are to be removed.

Berlin update

The recent tests changes:

Includes changes of releases previously marked as pre-release:
Changes of: https://github.com/ethereum/tests/releases/tag/7.0.2
Changes of: https://github.com/ethereum/tests/releases/tag/v7.0.1

Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/

Test format

• Fillers
  From pre releases a new folder under GeneralStateTests - VMTests containing reworked VMTests into state format

• Filled
  State tests post section contains new field txbytes with a transaction rlp. So no need to build a transaction from arrays and do the signing operation and to avoid confusion with transaction indexing. This field is yet to appear in new tests with the test regeneration.

Test Regeneration

Update all:

• Migrate VMTests in StateTests format
  The VMTests are being reviwed and reworked and now translated by Ori into StateTest format
  PR #791

• EIP-2315 (https://eips.ethereum.org/EIPS/eip-2315)
  Canceled. Rewoking the tests.
  PR #798
  PR #796

• Close issue #581 More EXTCODEHASH Tests
  PR #799

• Close issue #348 Revert after log operation
  PR #794
  PR #795

• Quick stack tests update:
  PR #790

Upcoming changes:

• old VMTests and it's blockchain form are to be removed.

AccessLists update

The recent tests changes:

Access lists tests, Underflow tests, documentation update
reminder of a web interface for test debugging: http://retesteth.ethdevops.io/web/

!!! quick fix for failing stack tests !!!
#790

Test format

• Fillers
  accessList support in transaction in generalState test fillers and blockchain test fillers.

• Filled
  in General State Tests transaction is expanded with optional field "accessLists" which size is 1 to 1 to "data" array. meaning each tx.data could have an accessList. {} - empty access list null - absent access list (legacy transaction) {access list json} - access list as defined by geth t8ntool. chainID is assumed 1 everywhere.

• In blockchain test transaction could have accessList. rlp of transaction is 01 + rlp (chainid, .... tx data fields) as described by EIP2930

• new test folder stEIP2930 in GeneralStateTests and BlockchainTests/GeneralStateTests

Test Regeneration

Update:
#774 - EIP2930 tests
#786 - Underflow tests
#788 - Documentation update
#784 - Documentation update

Upcoming changes:

• GeneralStateTest filled format update to include transaction rlp so that would be no need to calculate the signature from secret key. (#772)
• old VMTests and it's blockchain form are to be removed.

YoloV3 update

The recent tests changes:

EIPS covered with Berlin test configuration (geth YOLOv2, YOLOv3)

Test format

• The VMTests are being reworked as subSuite of GeneralStateTests.
  Same subsuite in BlockchainTests/GeneralStateTests/VMTests

• Fillers
  gtest fillers now support ":label somelabel " prefix in tx.data
  in expect section it is also possible to select transaction data indexes by label
  Filled
  tx.index -> label map is exported in gtest "_info" section for debug. is optional to parse it or not.

Test Regeneration

Update all:
PR #723
PR #775

• EIP2929 (https://eips.ethereum.org/EIPS/eip-2929)
  Gas cost increases for state access opcodes
  PR #760
  PR #756
  PR #745

• EIP2537 (https://eips.ethereum.org/EIPS/eip-2537)
  Precompile for BLS12-381 curve operations
  PR #713
  REMOVED: #776

• EIP2315 (https://eips.ethereum.org/EIPS/eip-2315)
  Simple Subroutines for the EVM
  PR #693
  PR #685

• EIP2565 (https://eips.ethereum.org/EIPS/eip-2565)
  ModExp Gas Cost
  existing test regeneration will test out the changes
  PR: #775
  PR: #779
  PR: #777

• VMTests rework under GeneralStateTests
  PR: #771

Upcoming changes:

• EIP2718 - EIP2930 (https://eips.ethereum.org/EIPS/eip-2718, https://eips.ethereum.org/EIPS/eip-2930)
  (Typed Transaction Envelope, Optional access lists)
  New test format (access list test example): PR: https://github.com/ethereum/tests/pull/772/files
  Access list tests PR: #774

• old VMTests and it's blockchain form are to be removed.

v7.0.0 - Pre-Berlin HF Release

This is the last Ethereum tests release with a pre-Berlin HF state, so it provides some stable ground to upgrade client test runners against to be ready for a Berlin hardfork integration.

Tests Added

StateTests / BlockchainTests

EIP-2200

Added state tests (also as BlockchainTests) to validate the EIP-1706/EIP-2200 out of gas condition,
specifically aimed at validating less than or equals to the stipend handling, see GeneralStateTests/stSStoreTest/sstore_gasLeft.json, PR #649

DIV/SDIV/MOD/SDIV by Zero

Added state tests (also as BlockchainTests) checking DIV/SDIV/MOD/SDIV by zero, see GeneralStateTests/stSolidityTest/ByZero.json, PR #647

Stack Validity of SWAP

Added tests checking stack validity of SWAP, see GeneralStateTests/stStackTests/stackOverflowSWAP.json and [..]/stacksanitySWAP.json, PR #647

EXTCODEHASH Empty Account

Added more EXTCODEHASH state tests (also as BlockchainTests) of nonexistent and post suicide accounts, see GeneralStateTests/stExtCodeHash/callToNonExistent.json,
[..]/callToSuicideThenExtcodehash.json and [..]/createEmptyThenExtcodehash.json, PR #654

SELFBALANCE in different Call Types

Added a composite state test (also in BlockchainTests) for SELFBALANCE that will do all the exisitng test inside of CALL, DELEGATECALL, and CALLCODE calls,
see GeneralStateTests/stSelfBalance/selfBalanceCallTypes.json, PR #671

Retesteth Unit Tests

Introduced retesteth unitTests in BlockchainTests/InvalidBlocks/bcExpectSection/filling_unexpectedAccount.json, PR #676

Difficulty Tests

Added difficulty tests for EIP-2384 (Muir Glacier Difficulty Bomb Delay), PR #662

Keystore Tests

Added KeyStore test based on MyCrypto file in order to ensure that the generated file is compatible with MyCrypto and Metamask, PR #665

RLP Tests

Added RLP invalid tests for insufficient bytes, see RLPTests/invalidRLPTest.json, PR #614

Test Format Changes

BlockChainTests

LegacyTests

BlockChain tests up to ConstantinopleFix (aka Petersburg) now also have been moved to the [LegacyTests/] folder, the main test folders on this release only
contain the Istanbul tests, PR #648

VMTests -> BlockchainTests

VMTests have been converted to BlockchainTests and can now be found in BlockchainTests/ValidBlocks/VMTests/, PR #680

PostState Correction

On some LegacyTests postState has been corrected to postStateHash, you can use this repo search
to see which tests are affected. This might need some adoption of your test runner.

InvalidBlockChain Tests: Expected Block Exceptions

Major blockchain test refactoring, and regeneration on geth + retesteth, see files diff for an impression on the format changes (InvalidBlockchainTests). PR #672

Retesteth

• Add Transition genesis retesteth configurations to default geth configs, PR #646
• Updated default mining reward config for transition nets in retesteth configs to avoid retesteth error when filling the state tests, PR #650
• Use additional forks in retesteth configs. Additional forks used in TransitionTests and not automatically used in StateTests/BlockchainTests, PR #651
• Enable Istanbul in aleth's retesteth configs, PR #656
• Fix ripemd160 precompile name in retesteth configs, PR #659
• Fix ECADD and ECMUL precompiles in Istanbul retesteth config, PR #660
• Add ChainID to Istanbul retesteth config, PR #661
• Remove retesteth configs, configs are now stored in retesteth repo. PR #666

Test Fixes

• Reduce 50k bytes code in quadratic complexity tests to 20k bytes. Resolves issue #657, see GeneralStateTests/stQuadraticComplexityTest/Call[*]KbytesContract[*].json, PR #658 and PR #669
• Minor changes to the test filler format around field strictness, affected tests
  BlockchainTests/InvalidBlocks/bcExpectSection/filling_wrongStorage2.json,
  GeneralStateTests/stExtCodeHash/*, GeneralStateTests/stRandom/ (3),
  GeneralStateTests/stReturnDataTest/subcallReturnMoreThenExpected.json,
  GeneralStateTests/stStaticCall/static_call_value_inherit.json,
  GeneralStateTests/stTransactionTest/Opcodes_TransactionInit.json,
  PR #677
• Upgrade and regenerate invalid RLP blockchain tests, PR #673
• Fix test fillers affected by the CompareStates bug, affected tests
  BlockchainTests/ValidBlocks/bcStateTests/randomStatetest377.json,
  GeneralStateTests/stCallCreateCallCodeTest/Call1024PreCalls.json,
  GeneralStateTests/stChangedEIP150/Call1024PreCalls.json,
  GeneralStateTests/stCreate2/create2checkFieldsInInitcode.json,
  GeneralStateTests/stDelegatecallTestHomestead/Call1024PreCalls.json,
  GeneralStateTests/stRandom/randomStatetest349.json,
  GeneralStateTests/stRandom2/randomStatetest578.json,
  GeneralStateTests/stRefundTest/*,
  GeneralStateTests/stStaticCall/*,
  PR #678
• Fixed blockgaslimit in CALLBlake2f_MaxRouds, see GeneralStateTests/stTimeConsuming/CALLBlake2f_MaxRounds.json, PR #679
• Removed bcExpectSection in BlockchainTests from public tests, PR #684

v7.0.0-beta.1 - Istanbul Support

This is the first Ethereum tests release with broader Istanbul support, see the "Istanbul support" section for a list with relevant PRs.

This release also comes with some structural changes to the test folder layout, the most important ones being the introduction of a new separate LegacyTests test suite for state tests up to Constantinople, a new sub folder structure for BlockchainTests and a new separate suite GeneralStateTests/stTimeConsuming/ for time consuming tests. For further details please have a look at the "Test Format Changes" section.

Istanbul Support

EIP Test Support

• EIP-152: Blake2b F precompile, CALL and CALLCODE tests added for the standard unit test vectors for Blake2b F, PR #619
• EIP-1344: ChainID opcode, state tests added in PR #627
• EIP-1884: Repricing for trie-size-dependent opcodes, SELFBALANCE and SLOAD gas cost tests added in PR #627

Test Regeneration

• Regeneration of BlockchainTests/GeneralStateTests (hive tests), PR #632
• Updated TransactionTests/ to Istanbul, PR #633
• Updated state tests with latest lllc, PR #635
• Updated GeneralStateTests to Istanbul, PR #639

Constantinople/Petersburg Updates

The following tests touching Constantinople/Petersburg behavior have been added or updated since the last release:

• Added ConstantinopleFix (aka Petersburg) tests, PR #582
• Updated ByzantiumToConstantinople transition test to ByzantiumToConstantinopleFix
  (see BlockchainTests/TransitionTests), PR #583, PR #588
• New SAR, SHL, SHR combinations, PR #574

Test Format Changes

• New LegacyTests suite for BlockchainTests/GeneralStateTests and GeneralStateTests for HFs up to Constantinople (so not: ConstantinopleFix aka Petersburg), PR #623
• New subfolder structure for BlockchainTests with added folders for InvalidBlocks and ValidBlocks,
  PR #605
• New separate suite GeneralStateTests/stTimeConsuming/ for time consuming tests, PR #595
• Moved blockchain specific tests from GeneralStateTests to BlockchainTests, PR #590
• Old unmaintained RPC test scripts in RPCTests have been removed, PR #573

Retesteth / RPC

• Added Istanbul to Retesteth configuration, PR #638
• Updated Retesteth configurations, PR #634
• Default Retesteth genesis configurations, PR #625
• Added Retesteth configuration for Pantheon client, PR #622
• Autokill Geth threads in startGethThreads.sh, PR #613
• Added fork configurations to Geth config, PR #602
• Fixed BlockchainTests RPC issues, PR #594

General Test Additions

• More RLP invalid tests for non-optimal lengths (see RLPTests/), PR #612
• More tests on touching precompiles along REVERT usage (see GeneralStateTests/stRevertTest/*Touch*.json), PR #580, PR #610
• New sStore test with non-zero nonce on collision (see GeneralStateTests/stSStoreTest/InitCollisionNonZeroNonce.json), PR #578

Bug Fixes/Optimizations

• Corrected tests with a missing expect section, PR #624
• Refill of all BlockchainTests/GeneralStateTests/, PR #621
• Removed post sections with no post conditions, PR #618
• Corrected huge expect sections in state tests (see stAttackTest/, 'stQuadraticComplexity/'),
  PR #617
• Compressed huge state data in bcExploitTest tests, PR #616
• Removed ambiguous test in BlockchainTests/ValidBlocks/bcMultiChainTest/ChainAtoChainB_blockorder2.json, PR #615
• Fixed RevertPrecompiledTouch test, PR #609
• Converted blockchain specific state tests into BlockchainTests, PR #607
• Various test fixes, PR #603
• Fixes and updates to various state tests, PR #599
• Fixed 0x missing in transaction -> data in GeneralStateTests, PR #598
• Removed a test case from badOpcodes state tests (see GeneralStateTests/stBadOpcode/), PR #592
• Fixed gasLimit issue in GeneralStateTests, PR #590
• Consistently use 0x prefixes in RLPTests, PR #587
• Removed underspecified lotsOfBranches test (see BlockchainTests/bcTotalDifficultyTest/lotsOfBranches.json), PR #579
• Changed gasUsed to 0 in genesisBlockHeader for dataTx tests (see BlockchainTests/bcValidBlockTest/dataTx.json and dataTx2.json), PR #577

v6.0.0-beta.3

This is the third release, which can be used to reference a static snapshot of the tests. It adds a lot of tests for EIP-1052 (EXTCODEHASH) and makes a few small breaking changes. Note that this release does not represent full Constantinople test coverage, there are more Constantinople tests incoming.

Potentially breaking changes

• #547 requires your integration to understand the network ByzantiumToConstantinopleAt5, which is analogous to existing pseudo networks such as FrontierToHomesteadAt5 and EIP158ToByzantiumAt5.

• #557 changes the format of the rlp tests, byte strings are now prefixed with "0x"

EIP 1234

• #547 added a test for the difficulty changes. This adds the network ByzantiumToConstantinopleAt5, which is analogous to existing networks such as FrontierToHomesteadAt5 and EIP158ToByzantiumAt5.

EIP 1052

• #548 Added a test for EXTCODEHASH called on an account created in the same transaction
• #549 added a few more tests for EXTCODEHASH and EXTCODESIZE on accounts created in the same transaction
• #550 Checks that changes to an account's nonce, balance, or storage do not change the result of calling EXTCODEHASH
• #552 Added another EXTCODEHASH test, what if it's run on accounts which have self-destructed?
• #563 Another spectacularly detailed test case of EXTCODEHASH involving recursive calls and oog-induced revert
• #566 checks that EXTCODEHASH returns a different result when the code changes
• #568 adds a test for EXTCODEHASH; what if the contract being hashed self destructed while in the middle of a delegate call?
• #569 calls EXTCODEHASH on an empty contract

Misc

• #551 tests for a Constantinople bug the fuzzer found in geth
• #544 adds an RLP test, courtesy of the mana team
• #557 changes the format of the rlp tests, byte strings are now prefixed with "0x"
• #561 changed some of the tests. There were some changes to testeth which made the previous tests impossible to regenerate. They were refactored but not in a way which should break any client.
• #567 adds a Constantinople test Mana found when they failed to sync with Ropsten

v6.0.0-beta.2 Release CW 46

This is the second of a new regular series of test releases, which can be used to reference a static snapshot of the tests within your library. First release has been v6.0.0-beta.1, see release notes from this release for an initial summary on the state of Constantinople tests.

Constantinople Test Updates

• Added initial test cases for EXTCODEHASH EIP-1052, see PR #484
• More EXTCODEHASH tests, see PR #544
• New SSTORE state tests and blockchain tests where an external call is overwriting/colliding with new SSTORE gas calculation rules, see PR #535

Test Coverage

• New tests to cover cases where the result of an EVM opcode is written to a specified memory range and the result is shorter than the specified range, see PR #538

Library Changes

• Added .idea to .gitignore, see PR #546

Docs

Test generation docs have been consolidated and integrated in the central ReadTheDocs testing documentation.

We also updated outdated parts on this doc section (see PR #539), so it should in principle now be possible to follow the guide and end up with a working test creation setup. There might still be some glitches, please let us know or submit a PR on ethereum/tests to if you stumble over something.

Other changes:

• New list with clients using the library in README, see PR #537

v6.0.0-beta.1 Release CW 43

This is the first of a new regular series of test releases, which can be used to reference a static snapshot of the tests within your library. For a starter release notes will give an overall summary of the status of consensus Constantinople tests, a summary of recent noteworthy changes and an overview on release versioning. Subsequent releases will then provide a CHANGELOG towards the release before.

Constantinople Test Summary

• EIP-145 (Bitwise shifting): Tests for SAR, SHL and SHR in GeneralStateTests/stShift directory, blockchain tests analogue
• EIP-1014 (CREATE2): Various cases covered in GeneralStateTests/stCreate2 directory, blockchain tests analogue
• EIP-1052 (EXTCODEHASH): Tests not merged yet, open PR #484
• EIP-1283 (SSTORE): Dedicated tests in GeneralStateTests/stSStoreTest directory also covering Ropsten consensus issue cases, blockchain tests analogue, generally refilled state tests with new SSTORE gas metering rules in PR #511
• EIP-1234 (difficulty): New difficultyConstantinople.json file and regenerated difficultyRopsten.json files in the BasicTests directory, see PR #518

Library Changes

Be aware that the format of BlockchainTests recently changed with the introduction of a new field sealEngine (values: NoProof | Ethash), see related JSON Schema change or BlockchainTest format docs for reference.

This means that you can faster-execute NoProof based tests skipping block validation. These tests nevertheless doesn't provide reliable values for PoW-based block header fields any more (mixHash, nonce), so make sure that you don't rely on the correctness of these values for the tests to pass.

Versioning

Library releases will follow semantic versioning:

• major number updates every time there is a backwards incompatibility for test runners, eg~
  • new fork rules
  • backwards-incompatible change, like dropping the mixhash and setting sealEngine=NoProof (see these release notes)
• minor number updates when new tests are added that are backwards compatible
• patch number updates if an existing test is bugfixed

Here is an example how a follow-up release line could look like:

• v6-alpha - starting to implement EIPs, but not all EIPs are finalized
• v6-beta - all EIPs finalized, but some tests still in progress
• v6.0.0 - all desired coverage for Constantinople tests to be considered "done enough"
• v6.1.0 - more coverage added
• v6.2.0 - more coverage added
• v6.2.1 - bugfix on one of the tests
• v7.0.0 - change in test format output (if backwards incompatible)