-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scraper stalls on random websites #280
Comments
Can you check the logs and let us know what feed?
|
The last time it ran it crashed on |
Hm, I'll try and reproduce this evening
|
And if I have no luck with that I'll come home and try to get the crash to
occur on a test server
|
It seems to be dead 100%. Can't ssh into it. @m1cr0man, is m1cr0man.com down |
Occasionally the scraper will stall indefinitely on a website. If I go into the screen session and hit
ctrl+c
it kills something associated with the requests library and then continues to the next website.My guess would be that to reproduce this all you would need to do is open 100 or so random URLs in series and watch for one of them not to respond.
This is in turn stopping the stream page updating and also stalling the email summaries. It needs to be fixed
The text was updated successfully, but these errors were encountered: