Creates an XML-Sitemap by crawling a given site.
Clone or download
Latest commit d28cd3d Dec 7, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.editorconfig push new version Aug 8, 2017
.gitignore split sitemaps for google Feb 26, 2017
.travis.yml chore(travis): set up semantic-release Oct 29, 2018
LICENSE Initial commit May 17, 2015
README.md adjust readme Sep 10, 2018
index.js Adding two more options max-concurrency and respect-robots-txt (#30) Sep 9, 2018
index.test.js update dependencies Nov 9, 2017
package-lock.json 7.4.0 Dec 7, 2018
package.json 7.4.0 Dec 7, 2018

README.md

Sitemap Generator CLI

Travis David npm

Create xml sitemaps from the command line.

Generates a sitemap by crawling your site. Uses streams to efficiently write the sitemap to your drive. Is cappable of creating multiple sitemaps if threshold is reached. Respects robots.txt and meta tags.

Table of contents

Install

This module is available on npm.

npm install -g sitemap-generator-cli
# or execute it directly with npx (since npm v5.2)
npx sitemap-generator-cli https://example.com

Usage

The crawler will fetch all folder URL pages and file types parsed by Google. If present the robots.txt will be taken into account and possible rules are applied for each URL to consider if it should be added to the sitemap. Also the crawler will not fetch URL's from a page if the robots meta tag with the value nofollow is present and ignore them completely if noindex rule is present. The crawler is able to apply the base value to found links.

sitemap-generator [options] <url>

When the crawler finished the XML Sitemap will be built and saved to your specified filepath. If the count of fetched pages is greater than 50000 it will be splitted into several sitemap files and create a sitemapindex file. Google does not allow more than 50000 items in one sitemap.

Example:

sitemap-generator http://example.com

Options

sitemap-generator --help

  Usage: cli [options] <url>

  Options:

    -V, --version                           output the version number
    -f, --filepath <filepath>               path to file including filename (default: sitemap.xml)
    -m, --max-entries <maxEntries>          limits the maximum number of URLs per sitemap file (default: 50000)
    -d, --max-depth <maxDepth>              limits the maximum distance from the original request (default: 0)
    -q, --query                             consider query string
    -u, --user-agent <agent>                set custom User Agent
    -v, --verbose                           print details when crawling
    -c, --max-concurrency <maxConcurrency>  maximum number of requests the crawler will run simultaneously (default: 5)
    -r, --no-respect-robots-txt             controls whether the crawler should respect rules in robots.txt
    -h, --help                              output usage information

filepath

Path to file to write including the filename itself. Path can be absolute or relative. Default is sitemap.xml.

Examples:

  • sitemap.xml
  • mymap.xml
  • /var/www/sitemap.xml
  • ./sitemap.myext

maxConcurrency

Sets the maximum number of requests the crawler will run simultaneously (default: 5).

maxEntries

fine a limit of URLs per sitemap files, useful for site with lots of urls. Defaults to 50000.

maxDepth

Set a maximum distance from the original request to crawl URLs, useful for generating smaller sitemap.xml files. Defaults to 0, which means it will crawl all levels.

noRespectRobotsTxt

Controls whether the crawler should respect rules in robots.txt.

query

Consider URLs with query strings like http://www.example.com/?foo=bar as indiviual sites and add them to the sitemap.

user-agent

Set a custom User Agent used for crawling. Default is Node/SitemapGenerator.

verbose

Print debug messages during crawling process. Also prints out a summery when finished.

License

MIT © Lars Graubner