How to deal with ETIMEDOUT error and pending forever?

[fanzijian]

Hello,
Thanks for the nice robust crawler. But I got ETIMEDOUT error sometimes. Besides, some http request may stay in pending state and the request died after 5 minutes.

Thanks
fanzijian

[tibetty]

Please refer to my modification over js-crawler from https://github.com/tibetty/ya-js-crawler/blob/master/crawler.js, line#406 & line#450

[amoilanen]

Thanks @tibetty We may need to include the modifications from https://github.com/tibetty/ya-js-crawler/blob/master/crawler.js into the default branch of https://github.com/antivanov/js-crawler/ and then we can publish them to NPM. I will take a look

[amoilanen]

And thanks to @fanzijian for reporting the issue. Will try to get the fix published soon

[tibetty]

@antivanov I had customized it too much to fit into my own purpose. I will try to figure out some useful but not so disruptive features and integrate into this project.

[amoilanen]

Thanks @tibetty, please create a pull request if there is time and some changes to contribute to the default branch of the project