v1.0.6-beta
Pre-release
Pre-release
Completely whitelists the original URL from any qualifying checks. This will be used as a failsafe if, for example, a link is incorrectly not crawled because it is detected as an unqualified file-type. The user can then take the URL of that and crawl it separately. Assuming it is a valid HTML or FTP page, it will crawl it properly.
Fully "production ready" release will include a detailed README.md, but the crawler itself is production ready.