-
Notifications
You must be signed in to change notification settings - Fork 5.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
What happens when a reponse takes very long #141
Comments
You should wrap |
I am a bit confused. It seems as if you want to put a timeout there, while I'm talking about the crawling "get" (targeting the server). It would seem the place where you suggest is that whenever somehow getting an item from the queue taking too long, then it would gracefully end. Whereas I'm in the |
I put it at both places, and that seems to solve some issues. But now whenever the queue is empty it will try to stop the worker, and it will throw a I catch it, but still I get 2 messages PER worker at the end of the script (not so nice for wanting to save the logging):
|
I tried to add:
instead of
In order to prevent hanging from a server by introducing a
max_timeout
, but this seems to open up a lot ofCancelledErrors
(and a lot ofTask was destroyed but it is pending
). Any idea?The text was updated successfully, but these errors were encountered: