TypeError in data_received of client_proto

[Insoleet]

Long story short

I migrated from aiohttp 1.0 to aiohttp 2.0. It seems to work correctly but it looks unstable.

Expected behaviour

No critical error.

Actual behaviour

I got a critical error in data received :

ERROR:root:Exception in callback _QEventLoop.__notifier_cb_wrapper({7: <PyQt5.QtCore...x7f27e9543d38>, 26: <PyQt5.QtCore...x7f27d1ca5c18>, 27: <PyQt5.QtCore...x7f27d1c275e8>, 30: <PyQt5.QtCore...x7f27d1a9c048>, ...}, <PyQt5.QtCore...x7f27d1aa2dc8>, 106, <bound method..., bufsize=0>>>, ())
handle: <Handle _QEventLoop.__notifier_cb_wrapper({7: <PyQt5.QtCore...x7f27e9543d38>, 26: <PyQt5.QtCore...x7f27d1ca5c18>, 27: <PyQt5.QtCore...x7f27d1c275e8>, 30: <PyQt5.QtCore...x7f27d1a9c048>, ...}, <PyQt5.QtCore...x7f27d1aa2dc8>, 106, <bound method..., bufsize=0>>>, ())>
Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/3.5.2/lib/python3.5/asyncio/events.py", line 125, in _run
    self._callback(*self._args)
  File "/home/inso/.pyenv/versions/sakia-dev-35/lib/python3.5/site-packages/quamash/__init__.py", line 435, in __notifier_cb_wrapper
    callback(*args)
  File "/home/inso/.pyenv/versions/3.5.2/lib/python3.5/asyncio/selector_events.py", line 676, in _read_ready
    self._protocol.data_received(data)
  File "/home/inso/.pyenv/versions/3.5.2/lib/python3.5/asyncio/sslproto.py", line 505, in data_received
    self._app_protocol.data_received(chunk)
  File "/home/inso/.pyenv/versions/sakia-dev-35/lib/python3.5/site-packages/aiohttp/client_proto.py", line 160, in data_received
    self._tail += data
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'bytes'

Steps to reproduce

I'm not sure what is the cause of this bug, but by analyzing the code I found some weird incoherencies. Here I found the lines

        if self._tail:
            data, self._tail = self._tail, None
            self.data_received(data)

Where self._tail is set to None before calling self.data_received.
https://github.com/aio-libs/aiohttp/blob/master/aiohttp/client_proto.py#L125

And here, it seems more correct, with self._tail being set to empty bytes.

        if self._tail:
            data, self._tail = self._tail, b''
            self.data_received(data)

https://github.com/aio-libs/aiohttp/blob/master/aiohttp/client_proto.py#L141

Here are some things I found in the stack. The parser is HttpResponseParserC. The HttpParserC returns None instead of empty bytes sometimes : https://github.com/aio-libs/aiohttp/blob/master/aiohttp/_http_parser.pyx#L285

Your environment

Python 3.5.2, aiohttp latest version (2.0.4).

[fafhrd91]

this looks suspicious. could you post example code?

seems server you are trying to connect to upgrade connection without checking incoming headers

[fafhrd91]

should be fixed in master

[Insoleet]

I encoutered this issue again. This is not fixed in 2.0.5. Please repoen !

Probably a simple fix would be to do the following, but it seems pretty dirty :

if not self._tail:
    self._tail = data
else:
    self._tail += data

The error happens randomly and I cannot reproduce it using a sample code. On the stack, I can see that everytime its the C parser which is being used. It probably comes from the C parser, which returns None sometimes instead of empty bytes :
https://github.com/aio-libs/aiohttp/blob/master/aiohttp/_http_parser.pyx#L285

Here is a screenshot of the HTTP Operation provoking the error, directly from the stack :

[fafhrd91]

could you try 2.0 branch

[Insoleet]

I let my software run actively this morning and didn't encoutered any error despite forcing it to send many requests. Looks all good !

[fafhrd91]

ok, I will release new version later today

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.