Streaming.py crash Incomplete Read

[pchai]

I've been using tweepy streaming to collect tweets for nearly one month, and this is the second time I received this error. Does anyone know why I am getting this error? Actually, I saw this issue before(streaming.py crash), but it's closed. I am wandering why do I still get this error?
Thank you for your help!

Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "twitterstreamtomongodb.py", line 255, in run
self.stream.filter(track=[self.searchterm])
File "/Library/Python/2.7/site-packages/tweepy/streaming.py", line 229, in filter
self._start(async)
File "/Library/Python/2.7/site-packages/tweepy/streaming.py", line 172, in _start
self._run()
File "/Library/Python/2.7/site-packages/tweepy/streaming.py", line 117, in _run
self._read_loop(resp)
File "/Library/Python/2.7/site-packages/tweepy/streaming.py", line 165, in _read_loop
self.on_closed(resp)
File "/Library/Python/2.7/site-packages/tweepy/streaming.py", line 177, in on_closed
self._start(async)
File "/Library/Python/2.7/site-packages/tweepy/streaming.py", line 172, in _start
self._run()
File "/Library/Python/2.7/site-packages/tweepy/streaming.py", line 117, in _run
self._read_loop(resp)
File "/Library/Python/2.7/site-packages/tweepy/streaming.py", line 150, in _read_loop
c = resp.read(1)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 541, in read
return self._read_chunked(amt)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 586, in _read_chunked
raise IncompleteRead(''.join(value))
IncompleteRead: IncompleteRead(0 bytes read)

[Aaron1011]

@pchai: Are you still having this problem with the latest version of Tweepy?

[magsol]

As of version 2.2, I am still receiving this error intermittently.

[Aaron1011]

@magsol: Can you try with 2.3, and also the latest checkout from the repository?

[magsol]

I've been running the streaming API from 2.3 for the past 48 hours with no problems. Most crashes usually happen in that time, but occasionally they take longer to manifest. I'll report back here if it happens and then try the latest checkout from the repository, but so far so good.

[magsol]

Figures I spoke too soon: just crashed again with the same IncompleteRead error as before:

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/user/script.py", line 54, in _read
    stream.sample()
  File "/usr/local/lib/python2.7/dist-packages/tweepy/streaming.py", line 290, in sample
    self._start(async)
  File "/usr/local/lib/python2.7/dist-packages/tweepy/streaming.py", line 237, in _start
    self._run()
  File "/usr/local/lib/python2.7/dist-packages/tweepy/streaming.py", line 173, in _run
    self._read_loop(resp)
  File "/usr/local/lib/python2.7/dist-packages/tweepy/streaming.py", line 214, in _read_loop
    c = resp.read(1)
  File "/usr/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 592, in _read_chunked
    value.append(self._safe_read(amt))
  File "/usr/lib/python2.7/httplib.py", line 649, in _safe_read
    raise IncompleteRead(''.join(s), amt)
IncompleteRead: IncompleteRead(0 bytes read, 1 more expected)

Will use the latest version from the repository and report back.

[bbenne10]

I'm seeing this with 2.3 as well. It's very intermittent here, but it's occuring about every three days or so,.

[ToonAlfrink]

I'm getting this pretty quick, takes only 2 minutes for it to appear.

[pranay01]

Any updates on this issue guys? I am still facing this.

[Aaron1011]

I'm not really sure what's causing this, unfortunately. If I had to guess, it's a network error on your end, or possibly with some Twitter server.

[magsol]

I had a thought the other day: could this possibly be caused by re-implementing on_timeout within the subclass of StreamListener, but returning True instead of False? I noticed recently that, for whatever reason, the method was returning True, and since switching it to False (a week ago or so) I haven't encountered the error since.

[Aaron1011]

@magsol: The default on_timeout actually returns None. Unless False is explicitly returned, Tweepy will ignore timeouts and reconnect, which is generally what you want. I'm not sure why the would be affecting the IncompleteRead, though

[magsol]

Ah ok. It was just a theory; I haven't looked closely enough at the code,
evidently.

On Sat, Jun 7, 2014 at 7:05 PM, Aaron1011 notifications@github.com wrote:

@magsol https://github.com/magsol: The default on_timeout actually
returns None. Unless False is explicitly returned, Tweepy will ignore
timeouts and reconnect, which is generally what you want. I'm not sure why
the would be affecting the IncompleteRead, though

—
Reply to this email directly or view it on GitHub
#242 (comment).

[raphdg]

It happens a lot (120 times in 2 hours yesterday during a world cup match) when I receive many limit messages. I'm using v2.3.0.

[cbelden]

Any other updates? This started happening to me today too. Using v2.3.0.

[raphdg]

Ok, I noticed this happens when the CPU reaches 100% (200 tweets per second seems to achieve it on ec2 micro instances). I haven't seen the problem running it at the same rate on c3.large instances.

[Harmon758]

Duplicate of #237.
This seems to have been fixed with fae9091 (#498), but was reverted with 34af675, due to #501.
As for why this occurs, see #448.