A crash occurs during settle gas estimation

[kelsos]

I was running my tests on kovan, I updated on master and tried to settle the channels. at some point all my clients crashed with the following message.

2018-11-08 16:27:14 [debug    ] settle called                  [raiden.network.proxies.token_network] locked_amount=0 locksroot=0x0000000000000000000000000000000000000000000000000000000000000000 node=82641569 partner=504300c5 partner_locked_amount=0 partner_locksroot=0x0000000000000000000000000000000000000000000000000000000000000000 partner_transferred_amount=0 token_network=11115746 transferred_amount=0
Traceback (most recent call last):
  File "/home/kelsos/.virtualenvs/raiden/bin/raiden", line 11, in <module>
    load_entry_point('raiden', 'console_scripts', 'raiden')()
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/__main__.py", line 11, in main
    run(auto_envvar_prefix='RAIDEN')  # pylint: disable=no-value-for-parameter
  File "/home/kelsos/.virtualenvs/raiden/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/kelsos/.virtualenvs/raiden/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/kelsos/.virtualenvs/raiden/lib/python3.7/site-packages/click/core.py", line 1043, in invoke
    return Command.invoke(self, ctx)
  File "/home/kelsos/.virtualenvs/raiden/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/kelsos/.virtualenvs/raiden/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/kelsos/.virtualenvs/raiden/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/ui/cli.py", line 425, in run
    app = runner.run()
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/ui/runners.py", line 245, in run
    return self._start_services()
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/ui/runners.py", line 96, in _start_services
    app_ = run_app(**self._options)
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/ui/app.py", line 418, in run_app
    raiden_app.start()
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/app.py", line 143, in start
    self.raiden.start()
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/raiden_service.py", line 372, in start
    self._initialize_transactions_queues(chain_state)
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/raiden_service.py", line 588, in _initialize_transactions_queues
    self.raiden_event_handler.on_raiden_event(self, transaction)
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/raiden_event_handler.py", line 103, in on_raiden_event
    self.handle_contract_send_channelsettle(raiden, event)
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/raiden_event_handler.py", line 484, in handle_contract_send_channelsettle
    partner_locksroot,
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/network/proxies/payment_channel.py", line 228, in settle
    partner_locksroot=partner_locksroot,
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/network/proxies/token_network.py", line 988, in settle
    partner_locksroot,
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/network/rpc/smartcontract_proxy.py", line 165, in estimate_gas
    raise err
  File "/home/kelsos/development/github/raiden-network/raiden/raiden/network/rpc/smartcontract_proxy.py", line 154, in estimate_gas
    'from': to_checksum_address(self.jsonrpc_client.address),
  File "/home/kelsos/.virtualenvs/raiden/lib/python3.7/site-packages/web3/contract.py", line 1194, in estimateGas
    **self.kwargs
  File "/home/kelsos/.virtualenvs/raiden/lib/python3.7/site-packages/web3/contract.py", line 1488, in estimate_gas_for_function
    gas_estimate = web3.eth.estimateGas(estimate_transaction)
  File "/home/kelsos/.virtualenvs/raiden/lib/python3.7/site-packages/web3/eth.py", line 304, in estimateGas
    [transaction],
  File "/home/kelsos/.virtualenvs/raiden/lib/python3.7/site-packages/web3/manager.py", line 112, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32015, 'message': 'Transaction execution error.', 'data': 'Internal("Requires higher than upper limit of 80000000")'}

[raiden.ui.runners] distribution=source python_implementation=CPython python_version=3.7.1 raiden=0.15.2.dev64+g29cb8933.d20181108 system=Linux 64bit_ 4.18.16-arch1-1-ARCH x86_64
[raiden.network.rpc.client] available_nonce=84 client=Parity-Ethereum//v2.1.3-beta-18ddd7c-20181015/x86_64-linux-gnu/rustc1.29.0 node=82641569

[LefterisJP]

I will have a look at this. Does it also happen with geth for you in latest master? Also can you provide full logs and the database?

[kelsos]

I have a geth node running on rinkeby and the raiden client that uses it, seems to work without any issue, the kovan nodes on the other hand are pretty consistent on crashing

[palango]

@LefterisJP I guess this is somehow relates to the gas estimation:

• here some people say that it's a parity problem

[LefterisJP]

here some people say that it's a parity problem

Nah it's not a parity problem. My investigation shows this is just parity's way of showing that a transaction will always fail with estimate gas. A simple call to our Kovan token network to register more tokens which should always fail returns that error:

curl -X POST --header "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_estimateGas","params":[{"from": "0x40bC6AA6514D8b6a3E169e62816385721A594E92", "to":"0x40bC6AA6514D8b6a3E169e62816385721A594E92", "data": "0x4cf71a040000000000000000000000000000000000000000000000000000000000000000"}],"id":1}' http://parity.kovan.ethnodes.brainbot.com:8545
{"jsonrpc":"2.0","error":{"code":-32015,"message":"Transaction execution error.","data":"Internal(\"Requires higher than upper limit of 80000000\")"},"id":1}

We need to handle that. We only handled it for geth. Will make a small PR for it. Check here.

But then on the other hand why would settle fail? Was it called at a wrong moment? Was it called multiple times?

Please provide full logs and a DB.

[christianbrb]

@LefterisJP Has this been fixed with PR #2986?

[LefterisJP]

Hmm the crash itself yes. The reason why settle would have failed is something I would be interested in -- but it's perhaps @kelsos upgraded the code manually and opened the channels with a previous version and settled them with another one. That could lead to funny behaviour.

@kelsos Can you confirm if this is true?

[kelsos]

Yes I can confirm that this was due to me opening and closing the channels in Andre's matrix PR, (database v10) and then opening the client during settle with master that had a breaking database change (database v11).

[LefterisJP]

All right so closing this then.