New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Raiden crashes with attribute error exception #2931

Closed
CosminNechifor opened this Issue Oct 30, 2018 · 6 comments

Comments

Projects
None yet
5 participants
@CosminNechifor
Collaborator

CosminNechifor commented Oct 30, 2018

Problem Definition

Raiden crashes with attribute error AttributeError: 'NoneType' object has no attribute 'identifiers_to_paymentnetworks. This error also happens to @kelsos

I got the following stacktrace when restarting my node after 4 hours when it was oflline. I want to mention that before actually killing my node I closed all the channels, so this might be related to the settlement.

Stacktrace:

018-10-30 15:08:23 [debug    ] State change                   [raiden.raiden_service] node=d6e68536 state_change={"transaction_hash": "0x6024b36de6478e1e6a38eb07c76e4ada3aed385037e119698a8c667096e6fe4f", "token_network_identifier": "0xa5C9ECf54790334B73E5DfA1ff5668eB425dC474", "channel_identifier": 35, "block_number": 6610678, "_type": "raiden.transfer.state_change.ContractReceiveChannelSettled", "_version": 0}
2018-10-30 15:08:23 [debug    ] Raiden event                   [raiden.raiden_service] node=d6e68536 raiden_event={"token_address": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2", "token_network_identifier": "0xa5C9ECf54790334B73E5DfA1ff5668eB425dC474", "channel_identifier": 35, "participant": "0x10F4BC8CD156B4785f8F1f100C53C4cA0181E9b9", "_type": "raiden.transfer.events.ContractSendChannelBatchUnlock", "_version": 0}
2018-10-30 15:08:35 [debug    ] Restoring from snapshot        [raiden.storage.wal]
2018-10-30 15:08:35 [debug    ] Replaying state changes        [raiden.storage.wal] num_state_changes=0
Traceback (most recent call last):
  File "raiden-script.py", line 2, in <module>
  File "site-packages/raiden/__main__.py", line 11, in main
  File "site-packages/click/core.py", line 722, in __call__
  File "site-packages/click/core.py", line 697, in main
  File "site-packages/click/core.py", line 1043, in invoke
  File "site-packages/click/core.py", line 895, in invoke
  File "site-packages/click/core.py", line 535, in invoke
  File "site-packages/click/decorators.py", line 17, in new_func
  File "site-packages/raiden/ui/cli.py", line 425, in run
  File "site-packages/raiden/ui/runners.py", line 243, in run
  File "site-packages/raiden/ui/runners.py", line 97, in _start_services
  File "site-packages/raiden/ui/app.py", line 410, in run_app
  File "site-packages/raiden/app.py", line 143, in start
  File "site-packages/raiden/raiden_service.py", line 364, in start
  File "site-packages/raiden/tasks.py", line 179, in first_run
  File "site-packages/raiden/tasks.py", line 185, in _run_callbacks
  File "site-packages/raiden/raiden_service.py", line 545, in _callback_new_block
  File "site-packages/raiden/blockchain_events_handler.py", line 326, in on_blockchain_event
  File "site-packages/raiden/blockchain_events_handler.py", line 253, in handle_channel_settled
  File "site-packages/raiden/raiden_service.py", line 475, in handle_state_change
  File "site-packages/raiden/raiden_event_handler.py", line 101, in on_raiden_event
  File "site-packages/raiden/raiden_event_handler.py", line 395, in handle_contract_send_channelunlock
  File "site-packages/raiden/storage/restore.py", line 28, in channel_state_until_state_change
  File "site-packages/raiden/transfer/views.py", line 339, in get_channelstate_by_id
  File "site-packages/raiden/transfer/views.py", line 240, in get_token_network_by_token_address
AttributeError: 'NoneType' object has no attribute 'identifiers_to_paymentnetworks'
[22683] Failed to execute script raiden-script
--- Logging error ---
Exception ignored in: <bound method BaseFileLock.__del__ of <filelock.UnixFileLock object at 0x7fd49f40d908>>
Traceback (most recent call last):
  File "site-packages/filelock.py", line 333, in __del__
  File "site-packages/filelock.py", line 320, in release
  File "logging/__init__.py", line 1308, in info
  File "logging/__init__.py", line 1444, in _log
  File "logging/__init__.py", line 1454, in handle
  File "logging/__init__.py", line 1516, in callHandlers
  File "logging/__init__.py", line 865, in handle
  File "logging/handlers.py", line 75, in emit
  File "logging/__init__.py", line 918, in handleError
  File "/usr/local/lib/python3.6/traceback.py", line 104, in print_exception
  File "site-packages/raiden/log_config.py", line 127, in tracebackexception_format
  File "site-packages/raiden/log_config.py", line 112, in processor_wrapper
ImportError: sys.meta_path is None, Python is likely shutting down

This is the most recent log: https://gist.github.com/CosminNechifor/6822a513869656fff49859570ef496a3

Also I will provide an archive which will contain all my logs and the database(this also might be helpful to have an exact view of what happened with my node):
all the logs I have from testing.zip

Issue checklist

Read through Requirements for safe usage mentioned in our documentation and make sure your usage follows the requirements.

  1. Is your Ethereum node syncing and up to date with the blockchain?
    Yes

  2. Are you using the same Ethereum account with anything other than the Raiden node? Or are you running multiple Raiden nodes with the same Ethereum account?
    No

  3. Does your Ethereum account used by Raiden have sufficient ETH?
    Yes

  4. Did you alter or delete the local DB that exists by default at ~/.raiden?
    No

  5. Did you stop / restart your raiden node while having open channels and/or pending transfers? if so, please provide further information.
    Yes the node was restarted. I had to leave to the university and before leaving I have closed some of my channels. After getting back and restarting it, I got the stacktrace above.

  6. Did you stop / restart your Ethereum node while Raiden was running?
    No.

  7. Did you switch from one Ethereum node to another? example: switching from a local Ethereum node to an Infura one.
    No.

  8. Is Raiden REST API protected from public access?
    Yes.

System Description

Raiden version: raiden-v0.15.1.dev6+g14cba904-linux
Linux noname 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
/usr/bin/solc
solc, the solidity compiler commandline interface
Version: 0.4.25+commit.59dbf8f1.Linux.g++
/usr/bin/geth
Geth
Version: 1.8.15-stable
Git Commit: 89451f7c382ad2185987ee369f16416f89c28a7d
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.10.1
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go-1.10
@kelsos

This comment has been minimized.

Contributor

kelsos commented Oct 30, 2018

When testing on kovan with raiden-v0.15.1.dev6+g14cba904-linux during the day I attempted to leave all networks settle all my channels and then open new channels.

I waited with all my nodes online for them to settle, and as soon as the channels settled, one of the nodes crashed. After the initial crash when I would get the crash Cosmin mentioned above as soon as I would start the node. This would happen each time.

I tried to replicate the issue so I left the network for all of my nodes again. Again after the channels settled two nodes crashed. After restarting though each node would give a different error, but the initial crash seems to be the same. You can find the log here and the DBs here

@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Oct 30, 2018

We have seen this before at #2874

This happens to the node when it needs to call unlock onchain right after a settlement. It tries to construct the chain state from the WAL and it returns a None chainstate.

@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Oct 30, 2018

I will close this as duplicated of #2874

@hackaugusto

This comment has been minimized.

Collaborator

hackaugusto commented Nov 12, 2018

I believe I found the problem. The partner's balance proof was not available in the state_changes table.

The error manifested itself with a node which has two versions of the database (8 and 9), it's possible the corresponding message was received on a first run which used version 8, but the channel was closed on version 9. This is not a Raiden bug but improper usage.

@LefterisJP

This comment has been minimized.

Collaborator

LefterisJP commented Nov 12, 2018

The error manifested itself with a node which has two versions of the database (8 and 9), it's possible the corresponding message was received on a first run which used version 8, but the channel was closed on version 9. This is not a Raiden bug but improper usage.

A lot of our error seems to come from this. I have mentioned it multiple time in dev-raiden-testing that people should not be doing that but still it happened. Perhaps #3006 will help. Also the underlying reason #2984 happened was due to closing happening in one database and settling in another.

@hackaugusto

This comment has been minimized.

Collaborator

hackaugusto commented Nov 13, 2018

it's possible the corresponding message was received on a first run which used version 8, but the channel was closed on version 9. This is not a Raiden bug but improper usage.

This was not the case. The channel with id 35 was opened on the database version 9, and the balance proof was in the database.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment