Retrieve raw http headers to see rate limit

[dennisvdvliet]

Although I know I can do a call to the twitter api to retrieve my current rate limit status it would be nice to benefit from the fact that twitter sends your rate limit back on every request.

How can I access the raw http headers of a request?

[sferik]

Could you say more about your use case? Do you have a proposal for how you'd like to access the rate limit data? This would require some significant changes to the library, but I'd be willing to do it if you make a compelling case.

[dennisvdvliet]

Please see the attached screenshot
twitter headers

Twitter sends back headers starting with "X-" on every request. Currently all replies from the Twitter module have a method called attrs which holds all the fields the Twitter api sends as reply. So I would be great if you could do the following

@twitter = Twitter.verify_credentials()
@twitter_attrs = @twitter.attrs()
@twitter_headers = @twitter.headers()

This would be great for scheduling and/or throttling requests to the Twitter api.

[sferik]

That's what I'm asking. How would you use the headers to schedule or throttle requests? Could you show me how you'd like it to work in code?

I'm asking because I suspect the library already has the capability to handle your use-case. Thousands of people use this library—many of whom bump up against rate limits—and this is the first request for this feature. It's certainly possible that you have a novel use-case that would uniquely benefit from being able to access the headers directly. It is also possible that having this feature wouldn't solve a new problem, but would allow you to solve the problem more elegantly. Either of these would justify adding the feature you've requested. I just want to be sure at least one of them is true before putting in hours of work to add this. If, at the end of the day, the solution that accessing headers provides is worse than existing solutions to deal with rate limiting, then we have only added complexity.

[dennisvdvliet]

My use-case is the following, I want to analyze all my followers and their timelines to advise on the best time to tweet. I'm doing this in a background job with resque.

So in the background I would perform the following tasks

• get all my followers
• get all their timelines

If I have 3500 followers this would take around 10 hours since I can only request 350 timelines per hour. I could determine the schedule at the beginning of the job based but I think the following approach would be nice and more fool proof

@friend_timeline = Twitter.user_time_line(:user_id => user_id)

# Save friend_timeline

if @friend_timeline.rate_limt['remaining_hits'] > 0
   Resque.enqueue_in('1 second', 'get next friends timeline')
else
   Resque.enqueue_in(@friend_timeline.rate_limit['rate_limit_reset'], 'get next friends timeline')
end

In this case the first 350 users would be processed in around 6 minutes and after that nothing would be done for the remainder of the hour. I hope this made it a little bit clearer.

[sferik]

Would this work?

begin
   @friend_timeline = Twitter.user_time_line(:user_id => user_id)
   Resque.enqueue_in('1 second', 'get next friends timeline')
rescue Twitter::Error::BadRequest => error
   Resque.enqueue_in(error.ratelimit_reset, 'get next friends timeline')
end

[dennisvdvliet]

Yes, that would work perfectly. Is that already possbile?

[sferik]

Yes. :)

[dennisvdvliet]

Thanks for your help!