streaming.py crash

[scardig]

Downloaded streamwatcher.py from tweepy-examples. Sometimes, after a day of running, it crashes with the following (I commented out lines 117->119 in stremaing.py to catch the exception):

Traceback (most recent call last):
File "webapps/siti/mex/scripts/mystreamwatcher.py", line 68, in
myfunc()
File "webapps/siti/mex/scripts/mystreamwatcher.py", line 63, in myfunc
stream.filter(followed)
File "/home/user/webapps/siti/mex/scripts/tweepy/streaming.py", line 195, in filter
self._start(async)
File "/home/user/webapps/siti/mex/scripts/tweepy/streaming.py", line 160, in _start
self._run()
File "/home/user/webapps/siti/mex/scripts/tweepy/streaming.py", line 111, in _run
self._read_loop(resp)
File "/home/user/webapps/siti/mex/scripts/tweepy/streaming.py", line 140, in _read_loop
c = resp.read(1)
File "/usr/local/lib/python2.6/httplib.py", line 517, in read
return self._read_chunked(amt)
File "/usr/local/lib/python2.6/httplib.py", line 563, in _read_chunked
raise IncompleteRead(value)
httplib.IncompleteRead

[chucheng]

You can somehow "catch all exception" and set self.running = True so that it won't stop. But I second it. I notice similar bug as well. (Not sure why httplib.IncompleteRead was raised). Python bug?

[sclm]

This issue has been persistant for me as well with several apps that I've built using Tweepy. I've patched it in my latest one and been gracefully allowed to share it back - I've checked it in, and will submit a pull request if it's stable and doesn't reappear.

If you'd like to see it, you can check out my edit: sclm/tweepy@8fbbc84ea1abf0f6273fc87f07f5d1bb6c5b6f99

[chucheng]

Hi Scim,

I look your patch, and I have a little concern about it.
Based on your patch, I think the only chance to raise the IncompleteRead is:
c = resp.read(1)
When this is happen, the "length" is actually a dirty value.

Once you catch a IncompleteRead, the next line become:
data = resp.read(length)
if self.listener.on_data(data) is False:
self.running = False
Now "length" in a read may become a problem because length may not be a integer.

What do you think?

[chucheng]

The following is my solution:
def _read_loop(self, resp):
data = ''
while self.running:
if resp.isclosed():
break

        # read length
        length = ''
        while True:
            try:
                c = resp.read(1)
            except httplib.IncompleteRead:
                length = ''
                break 
            if c == '\n':
                break
            length += c
        length = length.strip()
        if length.isdigit():
            length = int(length)
        else:
            continue

        # read data and pass into listener
        try:
            data = resp.read(length)
        except httplib.IncompleteRead:
            continue #data is imcomplete

        if self.listener.on_data(data) is False:
            self.running = False

[sclm]

The concern I have with your solution is that it fails to report the issue, suppressing it completely. I only ever encountered the incomplete read, and from what I've been able to discern is occurs when trying to grab the data from a connection that Twitter has broken on their end or has timed out. I can't promise that's the root cause, but that has been consistent with everything I've seen.

Consequently the goal of my code was to get it to drop out and reset the stream completely. Reconnecting to Twitter and hitting the error logic for those cases.

I think a wise alteration would be to move the try....except block I added to be more inclusive, to the outside of the outer loop and convert the break to a pass. In this case it would prevent the entire application from failing, but that function would terminate and the _read function would handle it appropriately.

[Harmon758]

This seems to have been fixed with fae9091 (#498), but was reverted with 34af675, due to #501. See #237.