Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Segmentation fault when CPU load is high #311
I'm seeing wpull 1.2.3 on Python 3.4.3 repeatably crash when running it on a machine with high CPU load. I've observed this on both Ubuntu 14.04 and 15.10 (different machines; the 14.04 is on a Core i3 from 2011 and 15.10 on a 4-core 4790K).
You can probably reproduce this by running this web server: https://github.com/ludios/crawl-destroyer
and then starting a lot of crawls with
After about 10-30 minutes on a 4-core 4790K, I see at least one process crash with
It may take longer to crash on other processors. A lower or higher number of wpull processes may be optimal, but I think the load average needs to be > 25 for a good chance of a crash.
This happens both with and without cchardet installed.
This one might be tricky to track down because heap corruption is probably happening some time before the crash. A Mozilla person suggested I use http://rr-project.org/ to try to track it down, but that the overhead might not be acceptable for Python. I will continue investigating.