Understanding and debugging Nimbus EVM JSON tests

Ștefan Talpalaru edited this page Dec 12, 2018 · 14 revisions


The Nimbus JSON tests are taken from the official Ethereum test suite. To find out more about the contents of this test suite, please refer to the official documentation.

You can run all tests with nimble test. Based on our config, Nimble will write binaries to build/ - you can do this manually also, as in the following examples:

Run example:

mkdir -p build
nim c -r -o:build/decompile_smart_contract examples/decompile_smart_contract.nim

Run Ethereum JSON-based general state tests:

mkdir -p build
nim c -r -o:build/test_generalstate_json tests/test_generalstate_json.nim

Build and run an individual test (with tracing enabled):

nim c -r -d:chronicles_log_level=TRACE -o:build/test_generalstate_json tests/test_generalstate_json.nim tests/fixtures/GeneralStateTests/stAttackTest/ContractCreationSpam.json

(this particular test needs to be commented out of "tests/test_generalstate_failing.nim" before running it)


JSON structure

A JSON EVM test has the following structure (example is add0.json).

    "add0" : {
        "_info" : {
            "comment" : "",
            "filledwith" : "cpp-1.3.0+commit.6e0ce939.Linux.g++",
            "lllcversion" : "Version: 0.4.18-develop.2017.9.25+commit.a72237f2.Linux.g++",
            "source" : "src/VMTestsFiller/vmArithmeticTest/add0Filler.json",
            "sourceHash" : "dcc7fc8aebdc2d7334440cfe6c63172941b4164c1ba8c32897318ca0cdfb7a1c"
        "callcreates" : [
        "env" : {
            "currentCoinbase" : "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba",
            "currentDifficulty" : "0x0100",
            "currentGasLimit" : "0x0f4240",
            "currentNumber" : "0x00",
            "currentTimestamp" : "0x01"
        "exec" : {
            "address" : "0x0f572e5295c57f15886f9b263e2f6d2d6c7b5ec6",
            "caller" : "0xcd1722f2947def4cf144679da39c4c32bdc35681",
            "code" : "0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff01600055",
            "data" : "0x",
            "gas" : "0x0186a0",
            "gasPrice" : "0x5af3107a4000",
            "origin" : "0xcd1722f2947def4cf144679da39c4c32bdc35681",
            "value" : "0x0de0b6b3a7640000"
        "gas" : "0x013874",
        "logs" : "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
        "out" : "0x",
        "post" : {
            "0x0f572e5295c57f15886f9b263e2f6d2d6c7b5ec6" : {
                "balance" : "0x0de0b6b3a7640000",
                "code" : "0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff01600055",
                "nonce" : "0x00",
                "storage" : {
                    "0x00" : "0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffe"
        "pre" : {
            "0x0f572e5295c57f15886f9b263e2f6d2d6c7b5ec6" : {
                "balance" : "0x0de0b6b3a7640000",
                "code" : "0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff01600055",
                "nonce" : "0x00",
                "storage" : {

Important: As of July 2018, VMTests always use Homestead fork for opcode implementations (notably CALL) and gas prices. We should not rely on currentNumber.

The important fields are:

  • exec section:

    • code: the code being executed in isolation. (In production this is part of a transaction).
    • gas: usually 0x0186a0 (100000) the starting gas
  • gas field: the remaining gas after code execution. Here 0x013874 (79988)

  • pre section: the state before execution of the code by the VM

    • storage: if the account already stores data that we can retrieve with SLOAD or overwrite/reset with SSTORE
  • post section: the state after execution of the code by the VM. If there is no post section, the code is supposed to throw an EVM exception. After an EVM exception, gas is consumed but state is reverted.

    Implementation-wise computation.isError returns true and an error message is available in computation.error field (implementation as of July 2018)

Decompiling bytecode.

You can decompile EVM bytecode using Etherscan: https://etherscan.io/opcode-tool Pasting 0x6003600202600055 gives:

[1] PUSH1 0x03 
[3] PUSH1 0x02 
[4] MUL 
[6] PUSH1 0x00 

Alternatively you can use the following:

import ../nimbus/vm/code_stream, strformat

var c = newCodeStreamFromUnescaped("0x6003600202600055")

let opcodes = c.decompile()
for op in opcodes:
  echo &"[{op[0]}]\t{op[1]}\t{op[2]}"

# [1]     PUSH1   0x03
# [3]     PUSH1   0x02
# [4]     MUL
# [6]     PUSH1   0x00
# [7]     SSTORE
# [-1]    STOP

The number in bracket refers to the position after reading the opcode and its arguments (i.e. the value of the program counter).

add0 (0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff01600055) decompiles to:

[32] PUSH32 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 
[65] PUSH32 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 
[66] ADD 
[68] PUSH1 0x00 

Debugging output

Nimbus outputs several tiers of debug information using the Chronicles library.

These are roughly split into several layers:

  • Trace
  • Debug
  • Info
  • Notice
  • Warn
  • Error
  • Fatal

In code, these look like this:

debug "Some debugging stuff", currentValue = value

Chronicles allows several levels of control over output, and these can be controlled by compiler flags when building Nimbus. The Chronicles repo goes into depth about how these flags operate.

Selecting output


When compiling using Nim with debug, Chronicles defaults to displaying debug and above messages, however this can be overridden by passing -d:chronicles_log_level= with one of the above levels. For example, you can compile using Nim's debug, but show only info level messages with -d:chronicles_log_level=INFO.


Some terminals don't support colour output and in this case you will see lots of extra characters that makes it difficult to read messages. To turn colouring off, you can use: -d:chronicles_sinks=textlines[nocolors, stdout]

Testing in other clients


  • run a single JSON test with tracing enabled (by the logging level):
pytest -o log_cli=true --log-cli-level=NOTSET -k fixtures/GeneralStateTests/stAttackTest/ContractCreationSpam.json tests/json-fixtures/test_state.py --fork Homestead

Installation hints:

  • use a virtualenv
  • pip install -e .[dev] might fail with a dependency conflict (which you can see later on with pip check). You work around that by manually downgrading the offending packages, like this:
pip install pluggy==0.7.1
pip install idna==2.7
# as of 2018-12-06, you also need:
pip install -e ./trinity-external-plugins/examples/peer_count_reporter
  • upstream runs tests using tox, so start from there to get to the relevant pytest arguments:
tox -l
tox -e py36-native-state-homestead
pytest tests/json-fixtures/test_state.py --fork Homestead


./build/env.sh go test -v -run TestState/stAttackTest/ContractCreationSpam.json ./tests
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.