max-mapper / find-data Public

Notifications You must be signed in to change notification settings
Fork 1
Star 13

CLI utility to spider websites and extract links to data files

13 stars 1 fork Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
.gitignore		.gitignore
README.md		README.md
cli.js		cli.js
client.js		client.js
index.js		index.js
package.json		package.json
preload.js		preload.js
test.js		test.js

Repository files navigation

work in progress

to test:

cd find-data
npm run demo

TODO

maxDepth
filterUrl
getLinks
domainWhitelist
supportedMimeTypes
stripQuerystring
stripWWWDomain
ignoreWWWDomain
scanSubdomains
filterByDomain
userAgent
timeout
delay

About

CLI utility to spider websites and extract links to data files

Report repository

Releases

No releases published

Packages

No packages published

Languages

JavaScript 100.0%