New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Routing problem on one hop transfer between long running nodes #2662

Closed
CosminNechifor opened this Issue Oct 3, 2018 · 10 comments

Comments

Projects
None yet
4 participants
@CosminNechifor
Collaborator

CosminNechifor commented Oct 3, 2018

Problem Definition

Maybe I did something wrong, but I experienced this problem and it seem to never let go. This is happening on two of my long running nodes and the token is one of the tokens used for the scenario player. I tried to perform a payment manually and I ended up with an internal server error.

raiden git:(master) ✗ curl -i -X POST http://127.0.0.1:5001/api/1/payments/0xa7023dBbD7e7DA194087374D9711af2Bd695fcBF/0x43735D6C531d68Ac88f4aBCB80926Db7185dFFa7 -H 'Content-Type: application/json' --data-raw '{"amount": 1, "identifier": 42}'
HTTP/1.1 500 INTERNAL SERVER ERROR
mimetype: application/json
Content-Type: application/json
Content-Length: 16
Date: Wed, 03 Oct 2018 14:33:01 GMT

{"errors": [""]}% 

Log:
raiden-debug.log

Proof that the channel actually exists

channel_0x77e080beaa49866d2966ab116a339057b3a07a43_0x43735d6c531d68ac88f4abcb80926db7185dffa7

curl -i http://localhost:5001/api/1/channels/0xa7023dBbD7e7DA194087374D9711af2Bd695fcBF/0x43735D6C531d68Ac88f4aBCB80926Db7185dFFa7    
HTTP/1.1 200 OK
mimetype: application/json
Content-Type: application/json
Content-Length: 327
Date: Wed, 03 Oct 2018 14:59:03 GMT

{"channel_identifier": 2, "balance": 20, "total_deposit": 20, "partner_address": "0x43735D6C531d68Ac88f4aBCB80926Db7185dFFa7", "settle_timeout": 500, "token_address": "0xa7023dBbD7e7DA194087374D9711af2Bd695fcBF", "token_network_identifier": "0x445F77125705E6824D56EC7431C8951506D975b3", "reveal_timeout": 10, "state": "opened"}% 
 curl -i http://localhost:5002/api/1/channels/0xa7023dBbD7e7DA194087374D9711af2Bd695fcBF/0x77E080beAA49866d2966Ab116a339057B3a07A43
HTTP/1.1 200 OK
mimetype: application/json
Content-Type: application/json
Content-Length: 325
Date: Wed, 03 Oct 2018 15:00:46 GMT

{"settle_timeout": 500, "channel_identifier": 2, "token_network_identifier": "0x445F77125705E6824D56EC7431C8951506D975b3", "state": "opened", "token_address": "0xa7023dBbD7e7DA194087374D9711af2Bd695fcBF", "balance": 0, "partner_address": "0x77E080beAA49866d2966Ab116a339057B3a07A43", "reveal_timeout": 10, "total_deposit": 0}% 

I can reproduce it anytime on my machine.

System Description

Linux noname 4.15.0-34-generic #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
/usr/bin/solc                                                                                                                                         
solc, the solidity compiler commandline interface                                                                                                     
Version: 0.4.25+commit.59dbf8f1.Linux.g++                                                                                                             
/usr/bin/geth                                                                                                                                         
Geth                                                                                                                                                  
Version: 1.8.15-stable                                                                                                                                
Git Commit: 89451f7c382ad2185987ee369f16416f89c28a7d                                                                                                  
Architecture: amd64                                                                                                                                   
Protocol Versions: [63 62]                                                                                                                            
Network Id: 1                                                                                                                                         
Go Version: go1.10.1                                                                                                                                  
Operating System: linux                                                                                                                               
GOPATH=                                                                                                                                               
GOROOT=/usr/lib/go-1.10                                                                                                                               
89190f86cbe9b7cbfa78e10314ba1aafc3e9c663
@palango

This comment has been minimized.

Collaborator

palango commented Oct 4, 2018

@CosminNechifor this seems a bit off, as the pathfinding algorithm seems to return a non-existent channel. Can you set a breakpoint at the assertion and look which channel it is?

@CosminNechifor

This comment has been minimized.

Collaborator

CosminNechifor commented Oct 4, 2018

@palango did that and there was no channel returned, as if it did not existed at all.

@hackaugusto

This comment has been minimized.

Collaborator

hackaugusto commented Oct 4, 2018

@CosminNechifor the attached log has only one post request:

2018-08-20 09:50:34 [info ] 172.29.0.4 - - [2018-08-20 09:50:34] "POST /api/1/payments/0x0f114A1E9Db192502E7856309cc899952b3db1ED/0x02f4b6BC65561A792836212Ebc54434Db0Ab759a HTTP/1.1" 200 439 2.675792 [raiden.api.rest.pywsgi]

which doesn't correspond to the error reported, could you please double check the logging file?

@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Oct 8, 2018

@CosminNechifor ping. You have been asked for more information here. Can you please respond?

@CosminNechifor

This comment has been minimized.

Collaborator

CosminNechifor commented Oct 8, 2018

@CosminNechifor I am looking into it right now @LefterisJP

@palango palango self-assigned this Oct 8, 2018

@palango palango added this to the Red Eyes Testnet 13 milestone Oct 8, 2018

@palango

This comment has been minimized.

Collaborator

palango commented Oct 8, 2018

@CosminNechifor Any updates?

@CosminNechifor

This comment has been minimized.

Collaborator

CosminNechifor commented Oct 8, 2018

Yes I can reproduce it.

@palango

This comment has been minimized.

Collaborator

palango commented Oct 8, 2018

Can you give a bit more infos? How can you reproduce it, how can I reproduce it. Does it happen with 0.12 as well?

@CosminNechifor

This comment has been minimized.

Collaborator

CosminNechifor commented Oct 8, 2018

Version: latest (bb680626d798af6c520506f249de379a5a2457fa)

Accounts on which this happens:

  • 0x77E080beAA49866d2966Ab116a339057B3a07A43
  • 0x43735D6C531d68Ac88f4aBCB80926Db7185dFFa7

Steps on how I reproduce it:

  1. I usually start my nodes connected to a geth node
  2. I try to perform a direct payment between those two nodes, using one of the scenario tokens just because I already have a channel for that.

Token address: 0xa6cd9705E6f34d85fd52908D15170A9707C629f5

Client 1 ( Initiator: 0x77E080beAA49866d2966Ab116a339057B3a07A43 ) output:
0x77E080beAA49866d2966Ab116a339057B3a07A43.txt

Client 2 ( Target: 0x43735D6C531d68Ac88f4aBCB80926Db7185dFFa7 ) output:
0x43735D6C531d68Ac88f4aBCB80926Db7185dFFa7.txt

debug.log:
raiden-debug.log

@palango

This comment has been minimized.

Collaborator

palango commented Oct 11, 2018

The assertion is hit because the network graph in RouteState includes more edges than there are channels in the network. see https://gist.github.com/CosminNechifor/040320fb1f00f23325ab42738f377d91

It's not clear yet why this happens, and at the moment only Cosmin can reproduce that.

palango added a commit to palango/raiden that referenced this issue Oct 12, 2018

palango added a commit to palango/raiden that referenced this issue Oct 12, 2018

@palango palango removed the investigating label Oct 12, 2018

rakanalh added a commit that referenced this issue Oct 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment