Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeouts and Aborted Connections after running code for long times #617

Closed
tednopple opened this issue Jun 5, 2015 · 47 comments
Closed

Timeouts and Aborted Connections after running code for long times #617

tednopple opened this issue Jun 5, 2015 · 47 comments
Labels
Bug Duplicate Improvement Question

Comments

@tednopple
Copy link

@tednopple tednopple commented Jun 5, 2015

I am using Twitter API to scan through tweets and users, but because of Twitter's API limits, scanning thousands of users/tweets can take hours. Fine. However, when I run Twitter API code through Python using Tweepy, after a few hours of the code running the code gets interrupted and I get one of two error messages:

tweepy.error.TweepError: Failed to send request: ('Connection aborted.', error(54, 'Connection reset by peer'))

Another error:

tweepy.error.TweepError: Failed to send request: HTTPSConnectionPool(host='api.twitter.com', port=443): Read timed out. (read timeout=60)

It seems Twitter has a timeout limit that stops access after a certain point? Or not? Is this a Twitter API issue, or a Tweepy one?

This doesn't happen for small sizes, only when the code has to spend more than about an hour running. Of course one sees the problem with walking away from script that should take hours to run, only to come back and find that it's aborted partway through.

No one seems to know why this happens. Posted on several forums and spoken to several people..Could be a bug.

@jamesonwatts

This comment has been minimized.

@natejlong

This comment has been minimized.

@ThaWeatherman
Copy link

@ThaWeatherman ThaWeatherman commented Jul 22, 2015

Bump, since I am too QQ...if it helps, here is my stack trace from within IPython

@mustekito

This comment has been minimized.

@astreim

This comment has been minimized.

@eabanoz
Copy link

@eabanoz eabanoz commented Aug 20, 2015

Besides these errors, I have also gotten this error code;

RATE LIMIT - waiting 15 minutes...
Traceback (most recent call last):
File "C:/Users/eabanoz/Desktop/Python_Scripts/Tweepy_Followers.py", line 73, in
print e.message[0]["code"]
TypeError: string indices must be integers, not str
TweepError Failed to send request: ('Connection aborted.', error(10054, 'An existing connection was forcibly closed by the remote host'))

Process finished with exit code 1

@ThaWeatherman

This comment has been minimized.

@lennybronner

This comment has been minimized.

@scubjwu
Copy link

@scubjwu scubjwu commented Sep 22, 2015

I just got the same error two days ago. You can quickly reproduce the error by turning off your network connection. After briefly check the source codes of tweepy, I think the error is from request lib because the default timeout value in tweepy is set to be 60s. So just let your program sleep some time and make sure the network connection is Okay, and the codes should be able to be resume.

@jcapde

This comment has been minimized.

@tednopple
Copy link
Author

@tednopple tednopple commented Sep 22, 2015

SORRY. This was for a client and Tweepy (or whoever) apparently couldn't care less about responding to the issue, plus the project was compete, so I stopped following the thread.

Our workaround was to divide the tweets up into smaller batches. We ran the code several times at once using different keys and smaller batches, then join the files later.

For future projects I probably won't use Tweepy, due to lack of support. We were just in too deep to turn to another library this time.

Though, after reading the thread, it looks like it could be a connection issue. We were working on company server so maybe connection dropped every once in a while, but I seem to recall this happening on a home computer as well and someone with higher expertise saying this was not the case.

Solutions so far:
-@jcapde87 's Try/Except solution also seems like a good idea.
-Use another library (Twitter)
-Divide data/calls into smaller batches and combine results later.

:(

@scubjwu
Copy link

@scubjwu scubjwu commented Sep 22, 2015

Hi @tednopple. I am just curious why using smaller batches would help ;-) Is it because smaller batches will shorter your time to process the data, so that the codes won't be timed out? Actually my codes just fetch the data from twitter and store it into database, so the data processing time is negligible. My feeling is the time out error may be raised by the rate limitation problem. You could always try to catch the exception and let the codes resume after sleeping for a while.

@tednopple
Copy link
Author

@tednopple tednopple commented Sep 22, 2015

Yes, I think smaller batches just cut the processing time down. We were scanning followers. People with hundreds of thousands of followers caused the time out. So we divided the list to process less than 100,000 at a time

We wrote an exception to catch rate limits and wait the required 15 minutes. Despite this, the code would just abort all together with large data after hours of running. And the error was a Tweepy one.

@aspectlab

This comment has been minimized.

@akisxyz

This comment has been minimized.

@allo-
Copy link

@allo- allo- commented Mar 8, 2016

I've got another error message to add:

File "/usr/local/lib/python2.7/dist-packages/tweepy/binder.py", line 149, in execute
    raise TweepError('Failed to send request: %s' % e)
tweepy.error.TweepError: Failed to send request: [Errno 110] Connection timed out

Another wish:
Can you split the TweepError into more detailed Exceptions? Like TweepyTimeout(TweepError), so it can be caught for a retry?

@hcmbg
Copy link

@hcmbg hcmbg commented Apr 30, 2016

I was having this ReadTimeOutError issue. I resolved this issue by catching the TweepError. @allo- you can access the TweepError's attribute (reason, response, api_code) to more specifically handle TweepError. In the particular error of 'Failed to send request' (e.g., raised in line 189), TweepError object only has reason.

In the try except block, I added:

except TweepError as e:
if 'Failed to send request:' in e.reason:
print "Time out error caught."
time.sleep(180)
continue

I have a while loop that wraps around the try except block. So whenever this TweepError is raised, the code sleeps for 180 sec. and continue executing the while loop. You can test this by running the code without internet access. Within that 180 sec, re connect the internet, and it should work.

@zeeraktalat

This comment has been minimized.

@raiprabh
Copy link

@raiprabh raiprabh commented Jul 28, 2016

A quick query, can specifying a longer timeout in api.timout in Tweepy object help in decreasing the occurrence of this issue?

@sam-s

This comment has been minimized.

@neuhaus

This comment has been minimized.

@zeeraktalat

This comment has been minimized.

@sam-s

This comment has been minimized.

@kjoth

This comment has been minimized.

@ThaWeatherman

This comment has been minimized.

@kjoth

This comment has been minimized.

@neuhaus
Copy link

@neuhaus neuhaus commented Feb 21, 2017

A code example how to catch a timeout and re-establish connection etc would be great.

@KhoaDuongUQ

This comment has been minimized.

@greysonevins
Copy link

@greysonevins greysonevins commented Jun 19, 2018

Any updates on this error?

@KhoaDuongUQ

This comment has been minimized.

@vzts

This comment has been minimized.

@vzts

This comment has been minimized.

@lwahedi

This comment has been minimized.

@lwahedi

This comment has been minimized.

@davekaj

This comment has been minimized.

@t-duan
Copy link

@t-duan t-duan commented Sep 8, 2019

I tried to initiate api = tweepy.API(timeout=600) #default=60 every time after time.sleep(15*60). It seems to work.

@Kanav-119

This comment has been minimized.

@saydus
Copy link

@saydus saydus commented Sep 10, 2020

I wonder why Tweepy has not addressed this issue yet. It persists even now. Will try solve timing out myself when I reach the boundary.

@danielp3011

This comment has been minimized.

@Harmon758
Copy link
Member

@Harmon758 Harmon758 commented Jan 19, 2021

@jcapde @ZeerakW @Kanav-119 This seems to be for streaming (which should now be resolved with 68e19cc) while this issue seems to be about API methods.

@neuhaus @ZeerakW If you're encountering an issue with streaming that isn't resolved with 68e19cc, then feel free to create a new issue. Otherwise, this issue seems to be about API methods.

@ZeerakW @sam-s #675 involves streaming, while this issue seems to be about API methods.

@vzts @davekaj That's for streaming and should be unnecessary now with 68e19cc.
#675 and #797 also involve streaming, whereas this issue seems to be about API methods.

@Harmon758
Copy link
Member

@Harmon758 Harmon758 commented Jan 19, 2021

This seems to be for streaming (which should now be resolved with 68e19cc) while this issue seems to be about API methods.

@KhoaDuongUQ

@dheeman00
Copy link

@dheeman00 dheeman00 commented Mar 1, 2021

I am facing this error while collecting the followers' information. Any suggestion will be helpful.

@tinyKyuu
Copy link

@tinyKyuu tinyKyuu commented Mar 21, 2021

@dheeman00 I'm currently facing the same issue while collecting follower and friend lists for my thesis. By any chance, were you able to come up with a solution yet?

@dheeman00
Copy link

@dheeman00 dheeman00 commented Mar 21, 2021

@dheeman00 I'm currently facing the same issue while collecting follower and friend lists for my thesis. By any chance, were you able to come up with a solution yet?

Well one way to resolve the issue by running your extraction in a While True loop with a try and except block in it. In that way if you encounter any issues it will pause for a moment and continue the extraction again.

@tinyKyuu
Copy link

@tinyKyuu tinyKyuu commented Mar 21, 2021

@dheeman00 Yeah, I was able to get it working by now. Thanks tho. Unfortunately, now I'm facing a new issue that I haven't seen posted anywhere else before.

When I get this list of a person's follower_ids and then use the look_up users functions for 100 users at a time, both lists don't have the same amount of entries.

I figured out why by looking up every user_id individually. Apparently one out of 11k user ids seem to give an error. I did get the tweep error that the id does not match to a user. However, the issue is that when doing a user lookup for multiple ids , it does not return an error. In that case it only returns 99 lookup requests, resulting in a different length of lists.

Any idea why that is the case? Did you perhaps face the same issue at some point?

@dheeman00
Copy link

@dheeman00 dheeman00 commented Mar 21, 2021

@dheeman00 Yeah, I was able to get it working by now. Thanks tho. Unfortunately, now I'm facing a new issue that I haven't seen posted anywhere else before.

When I get this list of a person's follower_ids and then use the look_up users functions for 100 users at a time, both lists don't have the same amount of entries.

I figured out why by looking up every user_id individually. Apparently one out of 11k user ids seem to give an error. I did get the tweep error that the id does not match to a user. However, the issue is that when doing a user lookup for multiple ids , it does not return an error. In that case it only returns 99 lookup requests, resulting in a different length of lists.

Any idea why that is the case? Did you perhaps face the same issue at some point?

Could you post your code where you are facing this issue. I can give you a better understanding when I can see this issue.

But running followers is extraction and look up is will work out one after the another. As you can extract max of 15k id per 15 mins and in the case of look up it is just 100.

@Harmon758
Copy link
Member

@Harmon758 Harmon758 commented Jun 7, 2021

Closing this in favor of #1606

@Harmon758 Harmon758 added Bug Duplicate Improvement Question labels Jun 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Duplicate Improvement Question
Projects
None yet
Development

Successfully merging a pull request may close this issue.