GNIP Twitter Stream hangs indefinitely #1

rweald opened this Issue Feb 8, 2012 · 4 comments


None yet
2 participants

rweald commented Feb 8, 2012

This has been reported by several people. It is caused by a change to GNIP's API. In particular the way authentication works for powertrack streams.

@ghost ghost assigned rweald Feb 8, 2012


rweald commented Feb 8, 2012

This issue is currently being addressed and should be fixed in the next couple of weeks.


eriwen commented Feb 19, 2012

I think there are 2 parts to getting this working:

The first is that it seems Gnip is forcing consumers to use compression for the PowerTrack stream (you get an HTTP 406 if you don't) and secondly the separate auth call to get a session token is gone.

My commit there fixes part of the problem by specifying compressed JSON and using basic auth. At this point the stream is no longer hanging but the JSON parser is choking on the compressed input. I couldn't figure out how to factor that in without changing how it goes about parsing JSON and dealing with the data buffer.

I didn't want to make a pull request because I didn't fix the whole issue, but I'm hoping the commit diff will be of some use to you. Cheers.


rweald commented Feb 20, 2012

Thanks for your comment and the commit!

I think the gzip formatting can be handled by em-http-request which is the underlying request library that I am using. I will take a look in the next couple of days and let you know


rweald commented Feb 29, 2012

After much effort I have been unable to find a workaround to the ruby streaming gzip problem. As a result I think the only way to proceed is going to be re-write the underlying http stream using curb.

This is not a super simple task so it might take a few extra days to accomplish.

@rweald rweald closed this in 0f62753 Sep 18, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment