New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node crashes with new_netting_channel failed #2923

Closed
Dominik1999 opened this Issue Oct 30, 2018 · 10 comments

Comments

Projects
None yet
5 participants
@Dominik1999
Contributor

Dominik1999 commented Oct 30, 2018

Intro

I tried to join the WETH network after a previous crashed. When I clicked on the "join network" button my node crashed with the following error message.

Problem Definition

018-10-29 13:30:42 [debug    ] State change                   [raiden.raiden_service] node=2a7d58e6 state_change={"block_number": 6605554, "gas_limit": 8000029, "block_hash": "0xc3c1a322c93d4dc7d119dc448f0d282bc3a0c35958f0710cf7196145540d0b08", "_type": "raiden.transfer.state_change.Block", "_version": 0}
2018-10-29 13:30:43 [debug    ] Getting token list             [raiden.api.rest] node=2a7d58e6 registry_address=0xB9633dd9a9a71F22C933bF121d7a22008f66B908
2018-10-29 13:30:43 [info     ] 127.0.0.1 - - [2018-10-29 14:30:43] "GET /api/1/tokens HTTP/1.1" 200 182 0.002705 [raiden.api.rest.pywsgi] 
2018-10-29 13:30:43 [debug    ] Getting connection managers info [raiden.api.rest] node=2a7d58e6 registry_address=0xB9633dd9a9a71F22C933bF121d7a22008f66B908
2018-10-29 13:30:43 [info     ] 127.0.0.1 - - [2018-10-29 14:30:43] "GET /api/1/connections HTTP/1.1" 200 110 0.003362 [raiden.api.rest.pywsgi] 
2018-10-29 13:30:44 [debug    ] Calculated gas price for transaction [raiden.network.rpc.client] calculated_gas_price=6842397831 node=2a7d58e6 node_gas_price=10000000000
2018-10-29 13:30:44 [debug    ] send_raw_transaction called    [raiden.network.rpc.client] gasLimit=6400000 gasPrice=6842397831 node=2a7d58e6 nonce=1
2018-10-29 13:30:44 [critical ] new_netting_channel failed     [raiden.network.proxies.token_network] peer1=2a7d58e6 peer2=10f4bc8c
2018-10-29 13:30:44 [critical ] Unhandled exception when processing endpoint request [raiden.api.rest] node=2a7d58e6
Traceback (most recent call last):
  File "site-packages/raiden/network/rpc/smartcontract_proxy.py", line 77, in transact
  File "site-packages/raiden/network/rpc/client.py", line 414, in send_transaction
  File "site-packages/web3/eth.py", line 274, in sendRawTransaction
  File "site-packages/web3/manager.py", line 112, in request_blocking
ValueError: {'code': -32010, 'message': 'Insufficient funds. The account you tried to send transaction from does not have enough funds. Required 43791346118400000 and got: 9219772000000000.'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages/flask/app.py", line 1813, in full_dispatch_request
  File "site-packages/flask/app.py", line 1799, in dispatch_request
  File "site-packages/flask_restful/__init__.py", line 480, in wrapper
  File "site-packages/flask/views.py", line 88, in view
  File "site-packages/flask_restful/__init__.py", line 595, in dispatch_request
  File "site-packages/webargs/core.py", line 478, in wrapper
  File "site-packages/raiden/api/v1/resources.py", line 194, in put
  File "site-packages/raiden/api/rest.py", line 648, in connect
  File "site-packages/raiden/api/python.py", line 207, in token_network_connect
  File "site-packages/raiden/connection_manager.py", line 167, in connect
  File "site-packages/raiden/connection_manager.py", line 431, in _open_channels
  File "src/gevent/greenlet.py", line 849, in gevent._greenlet.joinall
  File "src/gevent/greenlet.py", line 865, in gevent._greenlet.joinall
  File "src/gevent/greenlet.py", line 312, in gevent._greenlet.Greenlet._raise_exception
  File "site-packages/gevent/_compat.py", line 47, in reraise
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "site-packages/raiden/connection_manager.py", line 342, in _join_partner
  File "site-packages/raiden/api/python.py", line 302, in channel_open
  File "site-packages/raiden/network/proxies/token_network.py", line 170, in new_netting_channel
  File "site-packages/raiden/network/proxies/token_network.py", line 202, in _new_netting_channel
  File "site-packages/raiden/network/rpc/smartcontract_proxy.py", line 85, in transact
raiden.exceptions.ReplacementTransactionUnderpriced: Transaction was rejected. This is potentially caused by the reuse of the previous transaction nonce as well as paying an amount of gas less than or equal to the previous transaction's gas amount
2018-10-29 13:30:44 [debug    ] REST API stopped               [raiden.api.rest] node=2a7d58e6

more logs here - https://gist.github.com/Dominik1999/4c12e0daa360fc1da50a1afb86de12c3

Issue checklist

Read through Requirements for safe usage mentioned in our documentation and make sure your usage follows the requirements.

  1. Is your Ethereum node syncing and up to date with the blockchain?
  2. Are you using the same Ethereum account with anything other than the Raiden node? Or are you running multiple Raiden nodes with the same Ethereum account?
  3. Does your Ethereum account used by Raiden have sufficient ETH?
  4. Did you alter or delete the local DB that exists by default at ~/.raiden?
  5. Did you stop / restart your raiden node while having open channels and/or pending transfers? if so, please provide further information.
  6. Did you stop / restart your Ethereum node while Raiden was running?
  7. Did you switch from one Ethereum node to another? example: switching from a local Ethereum node to an Infura one.
  8. Is Raiden REST API protected from public access?

System Description

Here add a detailed description of your system, e.g. output of the following script:

Linux dominik-ThinkPad-T450s 4.19.0-041900-generic #201810221809 SMP Mon Oct 22 22:11:45 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

/usr/bin/solc
solc, the solidity compiler commandline interface
Version: 0.4.25+commit.59dbf8f1.Linux.g++

Geth (I use the brainbot node via VPN Access)

raiden-v0.15.1.dev6+g14cba904-linux
@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Oct 30, 2018

Duplicate of #2924 -- you don't have enough funds

@LefterisJP LefterisJP closed this Oct 30, 2018

@Dominik1999

This comment has been minimized.

Contributor

Dominik1999 commented Oct 30, 2018

That's not me to judge whether you consider this as a bug or not, but

  • as a user using the UI I would expect that there pops up a message saying "You don't have enough funds", here the node crashed
  • in addition, for me as a user using the UI, I can not see how much funds are needed. I just see that my node crashes
@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Oct 30, 2018

@Dominik1999 you are actually right. It should. Let me check this.

@LefterisJP LefterisJP reopened this Oct 30, 2018

@LefterisJP LefterisJP self-assigned this Oct 30, 2018

@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Oct 30, 2018

Hmmm there is an additional exception in there. So this seems like the other issue which is also seen by some people with ReplacementTransactionUnderpriced.

raiden.exceptions.ReplacementTransactionUnderpriced: Transaction was rejected. This is potentially caused by the reuse of the previous transaction nonce
@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Nov 1, 2018

I think this should be solved also by #2943

@LefterisJP LefterisJP closed this Nov 1, 2018

@hackaugusto

This comment has been minimized.

Collaborator

hackaugusto commented Nov 4, 2018

I don't think #2943 fixed this, the original exception is complaining about lack of funds

@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Nov 4, 2018

The reason I closed it was that I thought the lack of funds would have been caught but due to the connection manager spawning multiple open channels we hit the nonce issue before the funds error having enough time to propagate. But I checked and that's not possible.

The lack of funds is not caught by the gas reserve check.

That is probably due to the codepath being in the connection manager spawning multiple _join_partner greenlets all of which are trying to create a new channel so there is a potential race there or the has_enough_gas_reserve() function is not operating as we expect.

If it's the race issue then perhaps a lock can help?

@LefterisJP LefterisJP reopened this Nov 4, 2018

@christianbrb christianbrb assigned palango and unassigned LefterisJP Nov 5, 2018

@christianbrb christianbrb added this to the Red eyes testnet 16 milestone Nov 5, 2018

@palango

This comment has been minimized.

Collaborator

palango commented Nov 5, 2018

The lack of funds is not caught by the gas reserve check.

From my understanding this should have been caught by:

raiden/raiden/api/python.py

Lines 192 to 195 in 89cba57

has_enough_reserve, estimated_required_reserve = has_enough_gas_reserve(
self.raiden,
channels_to_open=initial_channel_target,
)

However, I think the available balance is enough, but the client runs into #2921 .

From here we have three options:

  1. Fix the gas_reserve check to temporarily include our gas estimation or
  2. Fix the gas_estimation (see PR #2623) or
  3. Fail nicer in this case.

@christianbrb I'd vote for option 2, but you had doubts there. What do you think?

@palango palango removed the investigating label Nov 5, 2018

@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Nov 5, 2018

@palango I have a branch where I am looking at this. I don't think that this is not caught. I am trying to reproduce this in a test in the aforementioned branch and it is caught. I think it may be a race with two calls going into the gas reserve before the node's balance being deducted for either case so both succeed.

@palango

This comment has been minimized.

Collaborator

palango commented Nov 5, 2018

I think it may be a race with two calls going into the gas reserve before the node's balance being deducted for either case so both succeed.

That might be right, if two independent calls to join a network are made, this will currently not be handled properly by the gas_reserve calculation.

palango added a commit to palango/raiden that referenced this issue Nov 9, 2018

palango added a commit to palango/raiden that referenced this issue Nov 9, 2018

@palango palango added the minor bug label Nov 9, 2018

palango added a commit to palango/raiden that referenced this issue Nov 9, 2018

palango added a commit to palango/raiden that referenced this issue Nov 9, 2018

palango added a commit to palango/raiden that referenced this issue Nov 9, 2018

LefterisJP added a commit that referenced this issue Nov 9, 2018

LefterisJP added a commit that referenced this issue Nov 9, 2018

rakanalh added a commit to rakanalh/raiden that referenced this issue Nov 16, 2018

rakanalh added a commit to rakanalh/raiden that referenced this issue Nov 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment