Skip to content

Scrapes links from specified URLs (specific domains can be filtered out)

License

Notifications You must be signed in to change notification settings

Aero-Blue/link-scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

link-scraper

Scrapes links (a["href"] tags) within a given webpage URL, specific domains / links can be filtered out.

Configuration support added:

URLs - A list of urls to scrape links from

Filters - A list of domains / words to filter when scraping links

UserAgent - Browser header to avoid being blocked by many websites

UpdateFrequency - Time between each link / scrape cycle

About

Scrapes links from specified URLs (specific domains can be filtered out)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages