Daemon refuses to start after crash.

[durkmurder]

Problem Definition

After opening a channel and depositing funds, my node crashed and refuses to start after this. Attaching crash dump and logs.
Logs when crashed:

2020-07-07 12:00:16.151924 [critical ] Unhandled exception when processing endpoint request [raiden.api.rest] node=0xa6Af4c9e671a7E70ebA8fb4509a119471116D5BC
Traceback (most recent call last):
  File "flask/app.py", line 1950, in full_dispatch_request
  File "flask/app.py", line 1936, in dispatch_request
  File "flask_restful/__init__.py", line 468, in wrapper
  File "flask/views.py", line 89, in view
  File "flask_restful/__init__.py", line 583, in dispatch_request
  File "raiden/api/rest_utils.py", line 59, in decorated
  File "raiden/api/v1/resources.py", line 114, in patch
  File "raiden/api/rest.py", line 1383, in patch_channel
  File "raiden/api/rest.py", line 1146, in _deposit
  File "raiden/api/python.py", line 782, in set_total_channel_deposit
  File "raiden/network/proxies/payment_channel.py", line 114, in approve_and_set_total_deposit
  File "raiden/network/proxies/token_network.py", line 852, in approve_and_set_total_deposit
  File "raiden/network/proxies/token_network.py", line 902, in _approve_and_set_total_deposit
  File "raiden/network/rpc/client.py", line 1140, in estimate_gas
  File "raiden/network/rpc/client.py", line 912, in estimate_gas
  File "raiden/network/rpc/client.py", line 898, in estimate_gas
  File "raiden/network/rpc/client.py", line 690, in patched_contractfunction_estimateGas
  File "raiden/network/rpc/client.py", line 648, in estimate_gas_for_function
  File "raiden/network/rpc/client.py", line 642, in estimate_gas_for_function
  File "raiden/network/rpc/client.py", line 588, in patched_web3_eth_estimate_gas
  File "raiden/network/rpc/client.py", line 582, in patched_web3_eth_estimate_gas
  File "web3/manager.py", line 153, in request_blocking
ValueError: {'code': -32000, 'message': 'execution reverted'

logs from dump:

Traceback (most recent call last):
  File "raiden/ui/cli.py", line 617, in run
  File "raiden/ui/runners.py", line 96, in run_services
  File "raiden/raiden_service.py", line 481, in stop
  File "raiden/network/transport/matrix/transport.py", line 547, in stop
  File "raiden/network/transport/matrix/client.py", line 522, in set_presence_state
  File "raiden/network/transport/matrix/client.py", line 180, in _send
  File "matrix_client/api.py", line 691, in _send
matrix_client.errors.MatrixRequestError: 404: 404 page not found

Logs after refusing to start:

You are connected to the 'goerli' network and the DB path is: /home/.../raiden-goerli-alice/node_a6af4c9e/netid_5/network_5a5cf4a6/v26_log.db
2020-07-07 12:08:12.919606 [error    ] Error processing request       [raiden.api.rest_utils] errors=Service unavailable. Try again later. status_code=<HTTPStatus.SERVICE_UNAVAILABLE: 503>
2020-07-07 12:08:12.923280 [info     ] 127.0.0.1 - - [2020-07-07 15:08:12] "GET /api/v1/payments HTTP/1.1" 503 204 0.004619 [raiden.api.rest.pywsgi] 
2020-07-07 12:08:12.925611 [error    ] Error processing request       [raiden.api.rest_utils] errors=Service unavailable. Try again later. status_code=<HTTPStatus.SERVICE_UNAVAILABLE: 503>
2020-07-07 12:08:12.926821 [info     ] 127.0.0.1 - - [2020-07-07 15:08:12] "GET /api/v1/pending_transfers HTTP/1.1" 503 204 0.001338 [raiden.api.rest.pywsgi] 
2020-07-07 12:08:12.956083 [error    ] Error processing request       [raiden.api.rest_utils] errors=Service unavailable. Try again later. status_code=<HTTPStatus.SERVICE_UNAVAILABLE: 503>
2020-07-07 12:08:12.958444 [info     ] 127.0.0.1 - - [2020-07-07 15:08:12] "GET /api/v1/channels HTTP/1.1" 503 204 0.002583 [raiden.api.rest.pywsgi] 
2020-07-07 12:08:12.961280 [error    ] Error processing request       [raiden.api.rest_utils] errors=Service unavailable. Try again later. status_code=<HTTPStatus.SERVICE_UNAVAILABLE: 503>
2020-07-07 12:08:12.963353 [info     ] 127.0.0.1 - - [2020-07-07 15:08:12] "GET /api/v1/tokens HTTP/1.1" 503 204 0.002221 [raiden.api.rest.pywsgi] 
2020-07-07 12:08:12.965334 [error    ] Error processing request       [raiden.api.rest_utils] errors=Service unavailable. Try again later. status_code=<HTTPStatus.SERVICE_UNAVAILABLE: 503>
2020-07-07 12:08:12.966747 [info     ] 127.0.0.1 - - [2020-07-07 15:08:12] "GET /api/v1/connections HTTP/1.1" 503 204 0.001649 [raiden.api.rest.pywsgi] 
2020-07-07 12:08:13.295830 [info     ] Using Matrix server            [raiden.network.transport.matrix.utils] average_rtt=0.17932666666666666 server_ident=Synapse/1.10.1 server_url=https://transport.transport02.raiden.network
2020-07-07 12:08:14.773526 [info     ] Updating channel fees          [raiden.raiden_service] cap_mediation_fees=True channel=CanonicalIdentifier(chain_identifier=5, token_network_address=b'>\xa2\xa1\xfe\xd7\xfd\xef0\r\xa1\x9e\x97\t,\xe8\xfd\xf8\xbff\xa3', channel_identifier=3851) flat_fee=0 proportional_fee=4000 proportional_imbalance_fee=3000

Traceback after refusing to start

Traceback (most recent call last):
  File "raiden/ui/cli.py", line 617, in run
  File "raiden/ui/runners.py", line 21, in run_services
  File "raiden/ui/app.py", line 449, in run_app
  File "raiden/app.py", line 91, in start
  File "raiden/raiden_service.py", line 443, in start
  File "raiden/raiden_service.py", line 551, in _start_transport
  File "raiden/network/transport/matrix/transport.py", line 468, in start
  File "raiden/network/transport/matrix/transport.py", line 922, in _initialize_health_check
  File "src/gevent/greenlet.py", line 1002, in gevent._gevent_cgreenlet.joinall
  File "src/gevent/greenlet.py", line 1018, in gevent._gevent_cgreenlet.joinall
  File "src/gevent/greenlet.py", line 364, in gevent._gevent_cgreenlet.Greenlet._raise_exception
  File "gevent/_compat.py", line 65, in reraise
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "raiden/network/transport/matrix/transport.py", line 618, in immediate_health_check_for
  File "raiden/network/transport/matrix/transport.py", line 1314, in _maybe_create_room_for_address
  File "raiden/network/transport/matrix/client.py", line 553, in create_room
  File "raiden/network/transport/matrix/client.py", line 225, in create_room
  File "raiden/network/transport/matrix/client.py", line 180, in _send
  File "matrix_client/api.py", line 691, in _send
matrix_client.errors.MatrixRequestError: 400: {"errcode":"M_UNSUPPORTED_ROOM_VERSION","error":"User's homeserver does not support this room version"}

System Description

{
  "raiden": "1.1.0",
  "raiden_db_version": 26,
  "python_implementation": "CPython",
  "python_version": "3.7.7",
  "system": "Linux 64bit_ELF 5.3.0-59-generic",
  "architecture": "x86_64",
  "distribution": "bundled"
}

[durkmurder]

Deleting data folder didn't help, as soon as it syncs data from chain it crashes again and refuses to start

[ulope]

@durkmurder Please never delete the data directory. Doing so will make any open channels that have had payment made in them unusable and will most likely lead to loss of tokens in those channels.

Can you please post the complete command line you used to start Raiden (and ideally also the complete debug log)?

[durkmurder]

I have a backup of data dir. Deleted it with hope to recover :)
./raiden-v1.1.0-linux-x86_64 --keystore-path MyCryptoRopstenAlice --eth-rpc-endpoint https://goerli.infura.io/v3/{INFURA_ID} --network-id goerli --datadir raiden-goerli-alice --accept-disclaimer --gas-price fast --environment-type development --routing-mode private --api-address 127.0.0.1:5001

[ulope]

That looks fine.
Does the problem still occur now?

Two days ago we had some problems with our test transport servers (that are used when on testnets) which could explain what you were seeing.

[durkmurder]

Seems fixed at the moment. Thank you, I am closing this issue.