Simple Crawler
This is a simple website crawler that visits a given URL, searches for hrefs on that page, adds those to a list, visits those URLs, etc. until all pages on a website have been crawled. (April 2014)
elitwilliams/simple-crawler
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|