Streaming crashes upon IncompleteRead

[eduardoltorres]

Description

Application displays the error below with no apparent periodicity (approximately once per day):

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/http/client.py", line 551, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/lib/python3.8/http/client.py", line 518, in _read_next_chunk_size
    return int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/http/client.py", line 583, in _readinto_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/lib/python3.8/http/client.py", line 553, in _get_chunk_left
    raise IncompleteRead(b'')
http.client.IncompleteRead: IncompleteRead(0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/eduardoltorres/.virtualenvs/the-debug-ducky-al9U58uy/lib/python3.8/site-packages/urllib3/response.py", line 436, in _error_catcher
    yield
  File "/home/eduardoltorres/.virtualenvs/the-debug-ducky-al9U58uy/lib/python3.8/site-packages/urllib3/response.py", line 518, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/lib/python3.8/http/client.py", line 454, in read
    n = self.readinto(b)
  File "/usr/lib/python3.8/http/client.py", line 488, in readinto
    return self._readinto_chunked(b)
  File "/usr/lib/python3.8/http/client.py", line 599, in _readinto_chunked
    raise IncompleteRead(bytes(b[0:total_bytes]))
http.client.IncompleteRead: IncompleteRead(226 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/eduardoltorres/.virtualenvs/the-debug-ducky-al9U58uy/lib/python3.8/site-packages/tweepy/streaming.py", line 320, in _run
    six.reraise(*exc_info)
  File "/home/eduardoltorres/.virtualenvs/the-debug-ducky-al9U58uy/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/eduardoltorres/.virtualenvs/the-debug-ducky-al9U58uy/lib/python3.8/site-packages/tweepy/streaming.py", line 289, in _run
    self._read_loop(resp)
  File "/home/eduardoltorres/.virtualenvs/the-debug-ducky-al9U58uy/lib/python3.8/site-packages/tweepy/streaming.py", line 339, in _read_loop
    line = buf.read_line()
  File "/home/eduardoltorres/.virtualenvs/the-debug-ducky-al9U58uy/lib/python3.8/site-packages/tweepy/streaming.py", line 200, in read_line
    self._buffer += self._stream.read(self._chunk_size)
  File "/home/eduardoltorres/.virtualenvs/the-debug-ducky-al9U58uy/lib/python3.8/site-packages/urllib3/response.py", line 540, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/eduardoltorres/.virtualenvs/the-debug-ducky-al9U58uy/lib/python3.8/site-packages/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(226 bytes read)', IncompleteRead(226 bytes read))

Next steps

Start by looking at the relevant discussions from the following Tweepy issues:

• Handle IncompleteRead error in Stream tweepy/tweepy#237
• Streaming.py Crash on Incomplete Read Error when tweets are very high... tweepy/tweepy#448

[eduardoltorres]

Stall warnings from #2 don't change behaviour in logging. Next attempt should be: queue and threading.