A very powerful crawler that will bring you all data from a website and it will also crawl sites that is linked to this webpage on one way or another, you will get all data from the source page! This tool is built with GNU/Parallell so you will use all your cores to to speed things up and crawl sites.
git clone https://github.com/wuseman/wspider
cd wspider; chmod +x wspider
./wspider -u <url> -p <path to store data>
Thats it, happy spidering!
-
Use this spider with caution it is a very powerful crawler, please check so the site you mirroring allowing crawling.
-
wuseman takes no responsibility whatsoever for what users use this tool for, each user is responsible for their own actions.
If you have problems, questions, ideas or suggestions please contact us by posting to wuseman@nr1.nu
Visit our homepage for the latest info and updated tools