Error 503 with lot of follow IDs

[rmorandi]

Hi,

I'm trying to setup the TwitterAPI with about 200 keywords and 4500 user IDs, and everything works fine until I reach 1200 userIDs. When adding more accounts I receive a 503 error...

is it possible that even if passing parameters, the request is still made using the GET method?

the code:

opz['track'] = an array of 200 keywords
opz['follow'] = an array of 1100 userIDs
r = self.api.request('statuses/filter', {'track': opz['track'], 'follow': opz['follow']})

response with > 1100 userIDs

STATUS CODE:  503
HEADERS:  {'date': 'Mon, 18 Aug 2014 16:38:12 UTC', 'connection': 'close', 'content-length': '31', 'x-connection-hash': '***xyz***'}

response with 1100 userIDS

STATUS CODE:  200
HEADERS:  {'transfer-encoding': 'chunked', 'date': 'Mon, 18 Aug 2014 16:40:12 UTC', 'connection': 'close', 'content-type': 'application/json', 'x-connection-hash': '***xyz***'}

Thanks,
Riccardo

[geduldig]

Hi Riccardo,

The track and follow parameters require a comma-separated string. Like,

'track':'keyword1,keyword2,keyword3'

There may be other issues, but try that first and see if it helps.

Jonas

[rmorandi]

Hi Jonas,

you're right, I forgot to mention that the array is joined under csv before passing it to the api.request.
Actually the script is working fine but I can't add more than 1100 users.

[geduldig]

I wonder if you have reached some buffer size limit. Can you try using 1101 users, but substitute some users names with shorter names so the entire string is no larger than the string with 1100 users?

[rmorandi]

the exact limit seems to be:
157 keywords and 1181 accounts, if I add something to the query (keyword or account, or even spaces) I get the error.

the last working parameter size is: 13.376 byte

I'm trying to troubleshoot the problem under a new fork, maybe is something missing in the session.request?

[geduldig]

Twitter's documentation for statuses/filter says 400 keywords and 5000 accounts. That is why I think you are hitting some buffer limit. Something to test would be if you can get more than 157 keywords or 1181 accounts without going over 13376 bytes, and see if that works. That would confirm my suspicion. HTTP has no limit on the query string size, and I haven't found a limit published by Twitter. It's also possible that the requests library imposes a limit, but I don't see that either.

[rmorandi]

I try splitting some keywords (example from #protezionecivile to "#protezione" and "civile") and got 503 error after just two split. Using Tweepy I was able to get all the 4500 accounts, but that library didn't fit my needings, I'm also taking a look at the request library, maybe the problem is there

[rmorandi]

ok.... don't ask me why, but I solved using the following code:

opz['track'] = ",".join(self.searchterm)
opz['follow'] = ",".join(self.searchterm2)
query = {'track': opz['track'], 'follow': opz['follow']}
r = self.api.request('statuses/filter', [query])

instead of

opz['track'] = ",".join(self.searchterm)
opz['follow'] = ",".join(self.searchterm2)
r = self.api.request('statuses/filter', {'track': opz['track'], 'follow': opz['follow']})

[geduldig]

Well, that is odd. The params are fed directly into the requests library method. That method takes a dictionary, not a list. I'll take a closer look...

[jpschmetz]

I have the same problem as well - my old Perl library does the job on the same input but I get a 503 whenever I follow more than 1534 people (about 13261 bytes worth of follow parameters). Again, same oauth token work with another library so the 503 error is probably not what it advertises itself to be (twitter overloaded). It definitely has to do with the numbers of bytes of parameters. The "fix" suggested by rmorandi doesn't work for me

[rmorandi]

jpschmetz can u paste the api.request and a part of the sent parameters?

[jpschmetz]

api.request('statuses/filter', {'follow': '1,2,3,4,5....'}) -- checked how the framework processes the request and what is sent to Session.requests is OK but it somehow make twitter unavailable on the other side

works fine until the string gets too large (but not too large for my access).

[rmorandi]

did you already try with:

opz['follow'] = "1,2,3,4,5..."
query = {'follow': opz['follow']}
api.request('statuses/filter', [query])

it seems to be exactly the same problem

[geduldig]

I have confirmed the problem as well. Wrapping params inside a list does make the problem go away for me. Problem seems to be in the requests library, so I will continue looking there. But for now, if you change line 98 in TwitterAPI.py to

params=[params]

should make things work until I find a real fix.

[jpschmetz]

I changed the line in TwitterAPI but it creates another problem for me

File "./testsample.py", line 27, in
r = api.request('statuses/filter', query)
File "/usr/local/lib/python2.7/dist-packages/TwitterAPI/TwitterAPI.py", line 105, in request
proxies=self.proxies)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 422, in request
prep = self.prepare_request(req)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 360, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 294, in prepare
self.prepare_url(url, params)
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 380, in prepare_url
enc_params = self._encode_params(params)
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 85, in _encode_params
for k, vs in to_key_val_list(data):
ValueError: need more than 1 value to unpack

[geduldig]

Jean-Paul,
Are you using OAuth1 or OAuth2?

[jpschmetz]

pretty sure OAuth1 - is it possible to tell from the keys or how do I know for sure?

[geduldig]

Default is OAuth1. You would need to specify OAuth2. Sorry I have no quick fix. Still looking into it.

[geduldig]

I committed version 2.2.3 with the fix. Now HTTP POST requests always put parameters in the body of the request instead of in the query string. It should be good now.

[rmorandi]

Hi, I'm sorry but the new version gives always the same 503 error.
I also discovered that my suggested fix didn't work (twitter accept the parameters but didn't filter the keys and the accounts you send)

if you give me an e-mail address I'll send you my full query

Thanks,
Riccardo

[rmorandi]

update: I'm using this fork for now and works fine

https://github.com/rmorandi/TwitterAPI/blob/master/TwitterAPI/TwitterAPI.py

query = {'track': 'a,b,c... 157 keys', 'follow': '1,2,3... 4500 IDs', 'filter_level':'none', 'stall_warnings':'true'}
r = self.api.request('statuses/filter', query)

the only thing I changed is on line 87:89 and added some debug... maybe could help ;)

[geduldig]

Hi Riccardo,
Send your query to boxnumber03@gmail.com.
Thanks!
Jonas

[geduldig]

I have not been able to produce errors with the current version (2.2.3). I am closing this issue, but if anyone is still seeing errors please post. Thanks!