How to handle IncompleteRead errors while streaming

[TayZ3r]

Everytime I run my code for a long time, I get that type of error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/http/client.py", line 555, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/local/lib/python3.8/http/client.py", line 522, in _read_next_chunk_size
    return int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/http/client.py", line 587, in _readinto_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/local/lib/python3.8/http/client.py", line 557, in _get_chunk_left
    raise IncompleteRead(b'')
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/response.py", line 436, in _error_catcher
    yield
  File "/usr/local/lib/python3.8/site-packages/urllib3/response.py", line 518, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/local/lib/python3.8/http/client.py", line 458, in read
    n = self.readinto(b)
  File "/usr/local/lib/python3.8/http/client.py", line 492, in readinto
    return self._readinto_chunked(b)
  File "/usr/local/lib/python3.8/http/client.py", line 603, in _readinto_chunked
    raise IncompleteRead(bytes(b[0:total_bytes]))
http.client.IncompleteRead: IncompleteRead(2 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/tweepy/streaming.py", line 320, in _run
    six.reraise(*exc_info)
  File "/usr/local/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/tweepy/streaming.py", line 289, in _run
    self._read_loop(resp)
  File "/usr/local/lib/python3.8/site-packages/tweepy/streaming.py", line 339, in _read_loop
    line = buf.read_line()
  File "/usr/local/lib/python3.8/site-packages/tweepy/streaming.py", line 200, in read_line
    self._buffer += self._stream.read(self._chunk_size)
  File "/usr/local/lib/python3.8/site-packages/urllib3/response.py", line 540, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(2 bytes read)', IncompleteRead(2 bytes read))

I already try to edit streaming.py like here

I have no idea how to fix it, any help is welcomed

[ethankershner]

I have been having this same issue, it will be solved in the next release I believe. I think the fix has been pushed to the main branch already but I was having issues installing from the main branch, but I believe that's one option to fix this before the release of 4.0.

EDIT: I do wonder if we could fix the problem temporarily by using a prior tweepy version.

[jamsspinle]

I've encountered this problem quite a few times and came up with a work around. I created an asynchronous function that runs in the background and checks every so often if the stream is still running. If it's not, it will create another. It looks something like this:

async def listen():
  stream = #createStream, make sure to create the stream so it's async.
  
  while(True):
    if not stream.running:
      await asyncio.sleep(5)
      stream = #createStreamAgain

    else:
      await asyncio.sleep(5) 

The sleep statement immediately after the if statement may not be needed, but I added it incase the API got mad that another stream was created so quick. Hopefully this helps you.

[ethankershner]

I've encountered this problem quite a few times and came up with a work around. I created an asynchronous function that runs in the background and checks every so often if the stream is still running. If it's not, it will create another. It looks something like this:

async def listen():
  stream = #createStream
  
  while(True):
    if not stream.running:
      await asyncio.sleep(5)
      stream = #createStreamAgain

    else:
      await asyncio.sleep(5) 

The sleep statement immediately after the if statement may not be needed, but I added it incase the API got mad that another stream was created so quick. Hopefully this helps you.

Thanks so much for this, I will try it out shortly.

[Harmon758]

This should already be fixed in the master branch with the mentioned commit, 68e19cc.
See also the relevant issues closed with that commit: #237, #448.
In the future, please make sure you're using the latest development version when reporting issues.

[TayZ3r]

I already applied 68e19cc changes on my streaming.py file, I stil get the error as I mentioned

[ethankershner]

I already applied 68e19cc changes on my streaming.py file, I stil get the error as I mentioned

You should install from the main branch instead of applying the changes on your own, see the readme.md for instructions on how to do that. You'll also want to subclass Streaming instead of StreamListener because those classes have been merged in the master branch version of Tweepy.

[Harmon758]

Applying individual commits from the master branch to an older version is ill-advised, as there can likely be context missing from additional prior or subsequent commits on the master branch. Regardless, even if 68e19cc was applied to v3.10 in the relevant parts of the code, that traceback would not be possible, since the error being raised is handled with that commit.

Manually modifying Tweepy itself is not reproducible and will not be supported. Instead, as noted, if you really need a change that hasn't been released yet, you should install the development version on the master branch, but there are other breaking changes currently on the master branch that you'll need to take into consideration. Otherwise, you should see the referenced issues for the likely cause of this issue.

Feel free to create a new issue if you're still able to reproduce this using the master branch.