Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

爬虫任务终止释放未完全问题 #18

Closed
LeoHuang2015 opened this issue Jan 7, 2020 · 5 comments
Closed

爬虫任务终止释放未完全问题 #18

LeoHuang2015 opened this issue Jan 7, 2020 · 5 comments
Labels
bug Something isn't working

Comments

@LeoHuang2015
Copy link

环境:windows10, crawlergo 0.1.2, chrome
单个测试效果不错,准备进行批量爬取。
单进程跑了一天发现电脑卡死,看了下cpu资源耗尽,后台发现chrome进程一大堆。
应该是crawlergo任务结束了,部分chrome并没有正常关闭。

@Qianlitp Qianlitp added the bug Something isn't working label Jan 7, 2020
@Qianlitp
Copy link
Owner

Qianlitp commented Jan 7, 2020

已发布新版本,请试用下看还有不有问题。

@LeoHuang2015
Copy link
Author

辛苦,今天回家跑一晚上看看。

@LeoHuang2015
Copy link
Author

使用最新版本的,测试还是存在问题的,机器信息:i78核 3.6ghz 内存16g。
测试扫描一个比较大的网站,就出现了这个问题。测试可以使用比如 https://sports.qq.com

@LeoHuang2015
Copy link
Author

LeoHuang2015 commented Jan 9, 2020

虽然3个tab,但是在爬取时会开启很多进程 参考图片

@Qianlitp
Copy link
Owner

目前测试未复现,还是推荐使用linux系统运行,更加稳定更快,不会有残留进程。
windows下开个虚拟机,或者起一个docker,都要比直接在windows下运行要好一些。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants