Message handling error when sending lots of large requests

[numirias]

I'm working on a plugin which frequently updates a large number (thousands) of highlights from a thread. To reduce the RPC overhead I'm using nvim_call_atomic, along the lines of:

reqs = [('nvim_buf_clear_highlight', (buf, hl.id, hl.start, hl.stop)) for hl in highlights]
nvim.api.call_atomic(reqs, async=True)

This works fine until the data gets large and the requests become more frequent. Then the client sometimes dies with this error:

2018-03-11 03:03:07,633 [DEBUG @ session.py:_yielding_request:152] 4267 - yielding from greenlet <greenlet.greenlet object at 0x7f3586c3d178> to wait for response
2018-03-11 03:03:07,633 [DEBUG @ msgpack_stream.py:_on_data:55] 4267 - waiting for message...
2018-03-11 03:03:07,633 [DEBUG @ msgpack_stream.py:_on_data:60] 4267 - unpacker needs more data...
2018-03-11 03:03:07,635 [DEBUG @ msgpack_stream.py:_on_data:55] 4267 - waiting for message...
2018-03-11 03:03:07,635 [DEBUG @ msgpack_stream.py:_on_data:57] 4267 - received message: [1, 18446744073709551615, [1, b'Message type must be an integer'], None]
2018-03-11 03:03:07,635 [DEBUG @ async_session.py:_on_response:95] 4267 - received response: [1, b'Message type must be an integer'], None
2018-03-11 03:03:07,636 [WARNING @ async_session.py:_on_message:78] 4267 - Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neovim/msgpack_rpc/async_session.py", line 75, in _on_message
    self._handlers.get(msg[0], self._on_invalid_message)(msg)
  File "/usr/lib/python3.6/site-packages/neovim/msgpack_rpc/async_session.py", line 96, in _on_response
    self._pending_requests.pop(msg[1])(msg[2], msg[3])
KeyError: 18446744073709551615

The "missing key" here is 0xffffffffffffffff and earlier in the log it says "Message type must be an integer", so it appears some message seems to be incorrectly (de)serialized.

Is this a known limitation? Am I doing something obviously wrong? Is there a different recommended way of sending a lot of requests at once? RIght now, I am solving this issue by sending multiple batches of atomic calls instead of all at once, but I'm just using an artificial limit without knowing what's going on.

Also, I am aware I should probably reduce the number of calls anyway, but I feel like this should still be handled gracefully.

[justinmk]

Maybe #325 will help.

[numirias]

Thanks for the pointer. As mentioned, I have a solid workaround for myself (sending small batches instead), so it's not an urgent problem for me.

However, I can still reproduce the problem that with large RPC traffic I occasionally get mentioned error. I'll see if I can provide more information, but the bug is somewhat elusive.

[darthdeus]

I'm running into this with https://github.com/numirias/semshi, any workarounds?

[numirias]

Seems like this was an internal error that's now gone. See numirias/semshi#17 (comment)