-
Notifications
You must be signed in to change notification settings - Fork 255
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Official Twitter bug affects twarc users too (error 404) #75
Comments
@remagio do you want to give the catch-404s branch a try and see if it helps? I've merged the pyopenssl branch into master, so the catch-404s branch should include it too. |
Yes, I'm ready with it for next scheduled jobs keeping log traces of https://github.com/edsu/twarc/compare/catch-404s#diff-22aabc32493075291fbea730860bc3dbR340 |
Updates @edsu: I'm still running two VM, a catch-404s and a current, both work fine during last 24h on same searches and archive.py without getting 404. Looks like Twitter is solving with tricks till they solve their infrastructures. |
Ok. I guess I'd rather not add this 404 catching unless it's necessary. So I'm going to leave in a branch for now. |
I agree, better waiting official feedbacks about how they'll manage same or new return status code |
FYI @edsu 404 raised up again, obviously not with the catch-404 branch |
Ok, I'll merge it in. |
There is an official bug since a couple of week (Twitter API calls on all major platform but their apps too) get frequently 404 with unknown state. It's discussed too here:
https://twittercommunity.com/t/intermittent-404-responses-from-rest-api/46712
Despite apparent resolution by Twitter it started to raise again.
Probably users of Twarc too got it recently (requests.exceptions.HTTPError: 404 Client Error: Not Found), it looks like this:
If it happens retry your search again till you you get it without errors. It's not a permanent failure but an intermittent failure by Twitter.
@edsu proposed to introduce a catch&retry. I share this because my opinion is that it'll happen in the future too since they killed the Whale.
The text was updated successfully, but these errors were encountered: