A Go web crawler library, CLI program, and REST API that aims to automate the generation of sitemaps for any size website.
The CLI tool lets you run the bot locally, or on a server. You can use it to generate sitemaps locally and upload, or to schedule a crawl and generation with cron or the likes.
go get -u github.com/lukeheuer.org/sitemapbot/cmd
Sitemap will be exported to public/sitemap.txt. Currently only plain-text single sitemap files are generated (valid for <50k sitemap pages only).
Report.csv gives a report of all URLs fetched along with load times (sorted longest load time to shortest).
- Generate a general new sitemapbot.conf
note: You can optionally generate a config per domain like this:
sitemapbot -nc domain.tld - it will load these settings when invoked with domain in future like this
Edit sitemapbot.conf See config/config.go for details on config directives.
sitemapbotin dir with config to run with your settings.
If you'd like to build SitemapBot, it is currently built with support for go1.8 and up.