Rate limit support, multiple authentication, search using v1.1 api and more. #282
Conversation
- API now support switching between multiple authentication handler - Allow blocking when reaching rate limit. - Search is now using Twitter API v1.1 - 'Pagination' of timelines is now done properly with max_id parameter, applies to all endpoints supporting 'max_id'. Other endpoint that have 'page' in their argument list still use the old pagination iterator. - Fixed an issue with PageIterator that caused the iterator to make an extra call to the Twitter API beyond the provided page limit. - Add unit tests for all of the above.
- iterating over search result using cursor with proper pagination. - calling user_timeline endpoint with the results parsed from JSON to python dictionary, not to other python class object.
|
Not sure I will accept the multiple authentications. Seems like a hack around the API rate limits. When it comes to rate limits, I usually leave that to the application to deal with those problems. |
|
@joshthecoder I under your position regarding the multiple authentication. Regarding blocking on reaching rate limit - it's not the default in my implementation but rather an option for the use case where you actually do want to monitor the rate limit and wait for its replenishment. Since the http response object is not exposed to the user of tweepy, she cannot monitor the remaining calls / block without first getting an error from twitter. The code in this pull request handles this scenario. Two other issues that were addressed in this pull request are proper pagination of timelines and fix of an issue with PageIterator that caused extra call to Twitter API. Also I included examples of changing payload type and use of pagination. Please consider merging those. |
|
Fair points. I'll have to spend more time combing though the changes and see what can go in. Thanks. |
|
Can the search part be merged in as a separate pull request? |
|
@nirg , do you have any objection to splitting your pull requests according to feature changes (better yet, one pull request/branch per issue?) I think that would streamline the review and acceptance process. |
|
@joshthecoder wrote:
I agree that rate limit policy should be delegated to the client application. I had the same concerns when I first read the pull request. The ideal situation (for me anyway) is that Tweepy would provide the rate limiting info to the client and allow the client to deal with it -- but Tweepy itself should not implement the policy logic by default if at all possible. |
|
I agree with the others having multiple features / fixes in a single request isn't ideal. Thanks for sharing and I would be happy to at least see the max_id/since_id based |
| payload_type = 'search_result', payload_list = True, | ||
| allowed_param = ['q', 'lang', 'locale', 'rpp', 'page', 'since_id', 'geocode', 'show_user', 'max_id', 'since', 'until', 'result_type'] | ||
| allowed_param = ['q', 'geocode', 'lang', 'locale', 'result_type', 'count', 'until', 'since_id', 'max_id', 'include_entities', 'rpp', 'page', 'show_user', 'since'] |
There was a problem hiding this comment.
You should remove a couple of allowed_params: "page" causes tweepy to reply with an error, "rpp" and "show_user" are ignored but don't generate errors (at the moment, anyway..). "cursor" doesn't really make sense here, and is also ignored (but doesn't generate an error)
|
Hi @nirg do you have an idea what I am doing wrong? I am getting an error when trying to switch to the next auth: http://stackoverflow.com/questions/42059191/tweepy-multiple-auth-handler |
Added multiple features (sorry for not separating commits):