Repository navigation
Releases: gaambo/website-crawler-tool
Releases · gaambo/website-crawler-tool
Release list
v1.3.0
Added
- SEO Meta Check: New
seo-metacheck that extracts common SEO metadata (title, description, canonical, robots, Open Graph, Twitter cards, hreflang) intoseo-meta.csv. - Collect URLs Check: New
collectcheck to only collect URLs. - Sitemap Index Support: The
--sitemapoption now supports sitemap indexes (e.g.,sitemap_index.xml) and resolves nested sitemaps.
Changed
- CLI Output UI: Improved progress and report output with an interactive spinner and colorized counters when running in a TTY.
- ESM + Dev Runner: Project is now ESM (
"type": "module") and the development workflow usestsx(npm run dev).
v1.2.0
Added
- URL List Support: New
-l, --url-list <file>option to crawl a specific list of URLs without recursion.- URL List Parser: Support for JSON and CSV file formats containing URL lists.
- Non-Recursive Mode: When using URL lists, the crawler only processes the specified URLs and does not discover new ones.
- Redirect Check: New
redirectscheck that detects URL redirects and captures initial redirect status codes.- Redirect Status Capture: Reports the initial HTTP status code (301, 302, etc.) used for redirection.
- Redirect Chain Information: Captures original URL, final URL, status code, and redirect chain in CSV output.
Changed
- Async Check Support: Updated check architecture to support async functions for HTTP-level checks.
Documentation
- Add
AGENTS.mdfile.
v1.1.0
Changed
- Each check now has a
keyproperty (e.g.,headings,images) which is used for CLI selection and as the base for the CSV export file name. - The CLI
-k/--checkoption now accepts the check's key (e.g.,-k imagesor-k headings). - Output CSV files are now named after the check key (e.g.,
images.csv,headings.csv). - The
nameproperty of each check is now used for human-readable output only, not for CLI or file selection.
Removed
- Removed support for using the check
nameas a CLI selector. Only thekeyis now supported.
Documentation
- Updated documentation to clarify the use of check keys for CLI and output.