Skip to content

Commit

Permalink
Parse http1.1 chunks to find json
Browse files Browse the repository at this point in the history
Chunked data sent via HTTP1.1 by Twitter is not as straight forward to read,
chunks indicating size of tweets are present no matter what, so we find the
json start in the buffer and forget the rest for now.

(Second part of correction for userstream and soon all streams. Fix python-twitter-tools#190 python-twitter-tools#116)
  • Loading branch information
RouxRC committed Dec 27, 2013
1 parent 73efaca commit 50de35b
Showing 1 changed file with 7 additions and 2 deletions.
9 changes: 7 additions & 2 deletions twitter/stream.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,16 @@ def __iter__(self):
if not self.block or self.timeout:
sock.setblocking(False)
while True:
utf8_buf = self.buf.decode('utf8').lstrip()
pos = utf8_buf.find('{')
if pos != -1:
utf8_buf = utf8_buf[pos:]
self.buf = utf8_buf.encode('utf-8')
try:
utf8_buf = self.buf.decode('utf8').lstrip()
res, ptr = self.decoder.raw_decode(utf8_buf)
self.buf = utf8_buf[ptr:].encode('utf8')
yield wrap_response(res, self.handle.headers)
if isinstance(res, dict):
yield wrap_response(res, self.handle.headers)
self.timer = time.time()
continue
except ValueError as e:
Expand Down

0 comments on commit 50de35b

Please sign in to comment.