New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON related error crashes node when matrix sends for invite user #2603

Closed
rakanalh opened this Issue Sep 25, 2018 · 2 comments

Comments

Projects
None yet
4 participants
@rakanalh
Contributor

rakanalh commented Sep 25, 2018

Problem Definition

Faced this issue today on one of my nodes:

You are connected to the 'ropsten' network and the DB path is: /home/rakan/.raiden/node_fc57d325/netid_3/network_f2a175a5/v4_log.db
2018-09-25 16:00:52 [info     ] Automatically selecting matrix homeserver based on RTT [raiden.network.transport.matrix] homeserver=https://transport01.raiden.network rtt=0.03135833333333333
2018-09-25 16:01:09 [debug    ] Restored state from WAL        [raiden.raiden_service] last_restored_block=4112435 node=fc57d325
Traceback (most recent call last):
  File "/home/rakan/.pyenv/versions/raiden/bin/raiden", line 11, in <module>
    load_entry_point('raiden', 'console_scripts', 'raiden')()
  File "/home/rakan/Brainbot/raiden/raiden/__main__.py", line 11, in main
    run(auto_envvar_prefix='RAIDEN')  # pylint: disable=no-value-for-parameter
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/core.py", line 1043, in invoke
    return Command.invoke(self, ctx)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/rakan/Brainbot/raiden/raiden/ui/cli.py", line 1193, in run
    NodeRunner(kwargs, ctx).run()
  File "/home/rakan/Brainbot/raiden/raiden/ui/cli.py", line 1023, in run
    app = self._run_app()
  File "/home/rakan/Brainbot/raiden/raiden/ui/cli.py", line 1042, in _run_app
    app_ = run_app(**self._options)
  File "/home/rakan/Brainbot/raiden/raiden/ui/cli.py", line 851, in run_app
    raiden_app.start()
  File "/home/rakan/Brainbot/raiden/raiden/app.py", line 124, in start
    self.raiden.start()
  File "/home/rakan/Brainbot/raiden/raiden/raiden_service.py", line 306, in start
    self.transport.start(self)
  File "/home/rakan/Brainbot/raiden/raiden/network/transport/matrix.py", line 202, in start
    self._join_discovery_room()
  File "/home/rakan/Brainbot/raiden/raiden/network/transport/matrix.py", line 467, in _join_discovery_room
    self._maybe_invite_user(user)
  File "/home/rakan/Brainbot/raiden/raiden/network/transport/matrix.py", line 1074, in _maybe_invite_user
    room.invite_user(user.user_id)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/matrix_client/room.py", line 327, in invite_user
    self.client.api.invite_user(self.room_id, user_id)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/matrix_client/api.py", line 503, in invite_user
    return self._send("POST", "/rooms/" + room_id + "/invite", body)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/raiden_libs/network/matrix/client.py", line 75, in _send
    return super()._send(method, path, *args, **kwargs)
  File "/home/rakan/.pyenv/versions/3.6.5/envs/raiden/lib/python3.6/site-packages/matrix_client/api.py", line 681, in _send
    errordata = json.loads(response.json()['error'])
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/rakan/.pyenv/versions/3.6.5/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

It seems to be that the error value:

    errordata = json.loads(response.json()['error'])

Is invalid JSON, causing the decoder to raise this exception.

@eorituz

This comment has been minimized.

Contributor

eorituz commented Sep 25, 2018

@andrevmatos related Traceback:

2018-09-25 17:15:59 [error    ] API DECODE: <Response [429]> '{"errcode":"M_UNKNOWN","error":"Unknown Status"}' [matrix_client.api] 
Traceback (most recent call last):
  File "/home/oliver/.virtualenvs/demo/lib/python3.6/site-packages/matrix_client/api.py", line 683, in _send
    errordata = json.loads(response.json()['error'])
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Btw: Forcing the Raiden node to use matrix02 works so far.

@andrevmatos

This comment has been minimized.

Collaborator

andrevmatos commented Sep 26, 2018

This was caused by a combination of server failure (transport03 crashed due to full disk, we're investigating how to prevent this from happening again) and matrix-python-sdk not handling the unexpected response properly. Upstream issue and PR to address this class of exceptions are referenced above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment