Failed Payment - Raiden crash

[junaidev]

Problem Definition

Simple operation of micro payment and querying payments failing and crashes raiden, its completely random.

2018-11-29 10:54:01 [info     ] 172.27.0.1 - - [2018-11-29 10:54:01] "GET /api/1/payments/0x4650D8E298C53948B60B05d56830E92a47480259/0x00285EFc12a22e1ea8Fd5F1b545FB3Ff95852E33 HTTP/1.1" 500 231 0.582140 [raiden.api.rest.pywsgi] 
FATAL: An unexpected exception occured. A traceback has been written to /tmp/raiden-exception-2018-11-29T10-543shpo8w8.txt
'EventPaymentSentFailed' object has no attribute 'initiator'
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "site-packages/raiden/ui/runners.py", line 209, in stop_task
  File "src/gevent/greenlet.py", line 633, in gevent._greenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 312, in gevent._greenlet.Greenlet._raise_exception
  File "site-packages/gevent/_compat.py", line 47, in reraise
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "site-packages/raiden/api/rest.py", line 435, in _run
  File "site-packages/gevent/baseserver.py", line 364, in serve_forever
  File "src/gevent/event.py", line 240, in gevent._event.Event.wait
  File "src/gevent/event.py", line 140, in gevent._event._AbstractLinkable._wait
  File "src/gevent/event.py", line 117, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/event.py", line 119, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
AttributeError: 'EventPaymentSentFailed' object has no attribute 'initiator'
2018-11-29T10:54:01Z <Greenlet "Greenlet-6" at 0x7f104809a348: stop_task(<raiden.api.rest.APIServer object at 0x7f104812e6d)> failed with AttributeError

Traceback (most recent call last):
  File "raiden-script.py", line 2, in <module>
  File "site-packages/raiden/__main__.py", line 11, in main
  File "site-packages/click/core.py", line 722, in __call__
  File "site-packages/click/core.py", line 697, in main
  File "site-packages/click/core.py", line 1043, in invoke
  File "site-packages/click/core.py", line 895, in invoke
  File "site-packages/click/core.py", line 535, in invoke
  File "site-packages/click/decorators.py", line 17, in new_func
  File "site-packages/raiden/ui/cli.py", line 425, in run
  File "site-packages/raiden/ui/runners.py", line 247, in run
  File "site-packages/raiden/ui/runners.py", line 214, in _start_services
  File "src/gevent/greenlet.py", line 849, in gevent._greenlet.joinall
  File "src/gevent/greenlet.py", line 865, in gevent._greenlet.joinall
  File "src/gevent/greenlet.py", line 312, in gevent._greenlet.Greenlet._raise_exception
  File "site-packages/gevent/_compat.py", line 47, in reraise
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "site-packages/raiden/ui/runners.py", line 209, in stop_task
  File "src/gevent/greenlet.py", line 633, in gevent._greenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 312, in gevent._greenlet.Greenlet._raise_exception
  File "site-packages/gevent/_compat.py", line 47, in reraise
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "site-packages/raiden/api/rest.py", line 435, in _run
  File "site-packages/gevent/baseserver.py", line 364, in serve_forever
  File "src/gevent/event.py", line 240, in gevent._event.Event.wait
  File "src/gevent/event.py", line 140, in gevent._event._AbstractLinkable._wait
  File "src/gevent/event.py", line 117, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/event.py", line 119, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
AttributeError: 'EventPaymentSentFailed' object has no attribute 'initiator'
[9] Failed to execute script raiden-script

and when restarting raiden node after this failure the node crashes immediately and node never starts again it keep on crashing with following:

usa_raiden_1_1  | Traceback (most recent call last):
usa_raiden_1_1  |   File "raiden-script.py", line 2, in <module>
usa_raiden_1_1  |   File "site-packages/raiden/__main__.py", line 11, in main
usa_raiden_1_1  |   File "site-packages/click/core.py", line 722, in __call__
usa_raiden_1_1  |   File "site-packages/click/core.py", line 697, in main
usa_raiden_1_1  |   File "site-packages/click/core.py", line 1043, in invoke
usa_raiden_1_1  |   File "site-packages/click/core.py", line 895, in invoke
usa_raiden_1_1  |   File "site-packages/click/core.py", line 535, in invoke
usa_raiden_1_1  |   File "site-packages/click/decorators.py", line 17, in new_func
usa_raiden_1_1  |   File "site-packages/raiden/ui/cli.py", line 425, in run
usa_raiden_1_1  |   File "site-packages/raiden/ui/runners.py", line 247, in run
usa_raiden_1_1  |   File "site-packages/raiden/ui/runners.py", line 96, in _start_services
usa_raiden_1_1  |   File "site-packages/raiden/ui/app.py", line 418, in run_app
usa_raiden_1_1  |   File "site-packages/raiden/app.py", line 152, in start
usa_raiden_1_1  |   File "site-packages/raiden/raiden_service.py", line 362, in start
usa_raiden_1_1  |   File "site-packages/raiden/tasks.py", line 163, in first_run
usa_raiden_1_1  |   File "site-packages/raiden/tasks.py", line 203, in _maybe_run_callbacks
usa_raiden_1_1  |   File "site-packages/raiden/raiden_service.py", line 547, in _callback_new_block
usa_raiden_1_1  |   File "site-packages/raiden/blockchain_events_handler.py", line 325, in on_blockchain_event
usa_raiden_1_1  |   File "site-packages/raiden/blockchain_events_handler.py", line 292, in handle_secret_revealed
usa_raiden_1_1  |   File "site-packages/raiden/raiden_service.py", line 477, in handle_state_change
usa_raiden_1_1  |   File "site-packages/raiden/raiden_event_handler.py", line 85, in on_raiden_event
usa_raiden_1_1  |   File "site-packages/raiden/raiden_event_handler.py", line 215, in handle_paymentsentsuccess
usa_raiden_1_1  | AssertionError
usa_raiden_1_1  | [10] Failed to execute script raiden-script
raiden_usa_raiden_1_1 exited with code 255

System Description

Here detailed description of system:

command -v solc && solc --version
/usr/bin/solc
solc, the solidity compiler commandline interface
Version: 0.4.23+commit.124ca40d.Linux.g++

command -v solc && solc --version
/usr/bin/solc
solc, the solidity compiler commandline interface
Version: 0.4.23+commit.124ca40d.Linux.g++

command -v geth && geth version
/usr/bin/geth
Geth
Version: 1.8.13-stable
Git Commit: 225171a4bfcc16bd12a1906b1e0d43d0b18c353b
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.10.3
Operating System: linux
GOPATH=
GOROOT=/home/travis/.gimme/versions/go1.10.3.linux.amd64

command -v raiden && raiden version
/usr/local/bin/raiden
{
  "raiden": "0.17.1.dev49+g27b4c34a.d20181129",
  "python_implementation": "CPython",
  "python_version": "3.6.7",
  "system": "Linux 64bit_ 4.15.0-39-generic x86_64",
  "distribution": "source"
}


/raiden# [ -d .git ] && git rev-parse HEAD
27b4c34a8eef067ab6a2fc3928fa775a643f3c59

[junaidev]

(this is happening when transfer is made using hub),
the hub node also crashes, on every restart with above scenario:

def sanity_check(state: MediatorTransferState) -> None:
    """ Check invariants that must hold. """

    # if a transfer is paid we must know the secret
    all_transfers_states = itertools.chain(
        (pair.payee_state for pair in state.transfers_pair),
        (pair.payer_state for pair in state.transfers_pair),
    )
    if any(state in STATE_TRANSFER_PAID for state in all_transfers_states):
        assert state.secret is not None

 Traceback (most recent call last):
usa_raiden_3_1  |   File "raiden-script.py", line 2, in <module>
usa_raiden_3_1  |   File "site-packages/raiden/__main__.py", line 11, in main
usa_raiden_3_1  |   File "site-packages/click/core.py", line 722, in __call__
usa_raiden_3_1  |   File "site-packages/click/core.py", line 697, in main
usa_raiden_3_1  |   File "site-packages/click/core.py", line 1043, in invoke
usa_raiden_3_1  |   File "site-packages/click/core.py", line 895, in invoke
usa_raiden_3_1  |   File "site-packages/click/core.py", line 535, in invoke
usa_raiden_3_1  |   File "site-packages/click/decorators.py", line 17, in new_func
usa_raiden_3_1  |   File "site-packages/raiden/ui/cli.py", line 425, in run
usa_raiden_3_1  |   File "site-packages/raiden/ui/runners.py", line 247, in run
usa_raiden_3_1  |   File "site-packages/raiden/ui/runners.py", line 96, in _start_services
usa_raiden_3_1  |   File "site-packages/raiden/ui/app.py", line 418, in run_app
usa_raiden_3_1  |   File "site-packages/raiden/app.py", line 152, in start
usa_raiden_3_1  |   File "site-packages/raiden/raiden_service.py", line 288, in start
usa_raiden_3_1  |   File "site-packages/raiden/storage/wal.py", line 47, in restore_to_state_change
usa_raiden_3_1  |   File "site-packages/raiden/transfer/architecture.py", line 261, in dispatch
usa_raiden_3_1  |   File "site-packages/raiden/transfer/node.py", line 1151, in state_transition
usa_raiden_3_1  |   File "site-packages/raiden/transfer/node.py", line 890, in handle_state_change
usa_raiden_3_1  |   File "site-packages/raiden/transfer/node.py", line 744, in handle_receive_unlock
usa_raiden_3_1  |   File "site-packages/raiden/transfer/node.py", line 180, in subdispatch_to_paymenttask
usa_raiden_3_1  |   File "site-packages/raiden/transfer/mediated_transfer/mediator.py", line 1440, in state_transition
usa_raiden_3_1  |   File "site-packages/raiden/transfer/mediated_transfer/mediator.py", line 233, in sanity_check
usa_raiden_3_1  | AssertionError
usa_raiden_3_1  | [10] Failed to execute script raiden-script

[LefterisJP]

@junaidev Thanks for the bug report.

'EventPaymentSentFailed' object has no attribute 'initiator'

As far as this error is concerned I know what's wrong. I am preparing a PR to fix it.

File "site-packages/raiden/transfer/mediated_transfer/mediator.py", line 233, in sanity_check

For this assertion in the sanity check I would need the entire logs from before the restart and after to see how come it came to this state for the secret to be None.

[LefterisJP]

@junaidev And preferrably for the second problem and the logs of it please make another issue as it's much easier for us to handle if we got one issue per problem.

[junaidev]

@junaidev And preferrably for the second problem and the logs of it please make another issue as it's much easier for us to handle if we got one issue per problem.

These issues are linked, as if a node was crashing due to ('EventPaymentSentFailed' object has no attribute 'initiator') same time the hub was also going down.

[LefterisJP]

If they are linked then the fix should handle it. If not then open another issue.