「 gargantua 」
The fast website crawler
You can use「 gargantua 」to quickly and easily
- warm-up your frontend caches
- perform small load-tests against your publicly available pages
- measure response times
- detect broken links
from your command line on Linux, macOS and Windows.
Qto stop the current crawling process.
Crawl www.sitemaps.org with 5 concurrent workers:
gargantua crawl --url https://www.sitemaps.org/sitemap.xml --workers 5
You can download binaries for Linux, macOS and Windows from github.com »andreaskoch » gargantua » releases:
There is also a docker image that you can use to download or run the latest version of gargantua:
docker run --rm andreaskoch/gargantua:latest \ crawl \ --verbose \ --url https://www.sitemaps.org/sitemap.xml \ --workers 5
Note: You will need the
--verbose flag in order to prevent the command-line UI from loading. Otherwise gargantua will fail.
- Increase the number of workers at runtime
- Personalized user agent string
- Silent mode (only show statistics at the end)
- CSV mode (print CSV output to stdout)
- Save downloaded data to disk
「 gargantua 」is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.