v3.0.0

The first stable 3.x release, now latest on npm. The 2.x line stays maintained on the v2 branch for bug and security fixes (npm install mapsite@2).

npm install mapsite

Changelog

The constructor, option names, run, fromBuffer and the { type, urls, errors }
result are unchanged. The internals were rewritten; see "Upgrading from 2.x" in the
README for the full list of differences.

Fixed

• Works on Bun: redirects are followed internally instead of via Dispatcher.compose.
• proxy works on Bun, whose built-in undici ignored ProxyAgent and sent
  requests directly; proxied requests now use Bun's fetch proxy option.
• A Content-Type header with several comma-joined values is accepted if any is allowed.

Known limitations

• On Bun, timeout is not enforced and close() does not close connections:
  Bun's built-in undici ignores them.

Breaking

• type reports the root document: a sitemap index is "index" (2.x returned
  whichever document finished last).
• Removed the debug option, SitemapParser.getLinesFromText, and the
  currentDepth, currentRetry and maximumDepth properties.
• maximumRetries defaults to 1 regardless of other options (2.x used 3 only
  without an options object) and accepts 0. Only transient failures are retried,
  with exponential backoff; 4xx responses other than 408, 425 and 429 are not.
• Content types are matched exactly instead of by substring.
• Invalid options throw a TypeError naming the option instead of a ZodError.
• Pages and sitemaps seen more than once are skipped; pages come back in document
  order.
• Documents over 50 MiB (compressed or decompressed) are rejected; see
  maximumResponseSize.
• <loc> is read only as a child of <url> or <sitemap>.

Added

• entries on the result, behind the includeEntries option (default false): lastmod, changefreq, priority and hreflang
  alternates next to each URL.
• discover(site) finds sitemaps through robots.txt (falling back to
  /sitemap.xml).
• stream(url) yields entries as they are found, and stops fetching when you stop
  iterating.
• AbortSignal support on every method, and close() / await using.
• Plain-text sitemaps, RSS and Atom feeds.
• Redirects are followed, and gzip is detected from the data rather than the
  content type.
• Options concurrency, maximumResponseSize and retryDelay.
• proxyMode: 'fallback': fetch directly and only use the proxy for sites that
  answer 401, 403 or 429.
• blockPrivateAddresses: refuse direct connections to non-public addresses,
  including through redirects and DNS, for crawling user-supplied URLs.
• ESM and CommonJS builds with an exports map.

Changed

• Rewrote the crawler and HTTP layer; sitemaps are fetched concurrently without the
  fixed three-at-a-time batches and delays.
• Parsing uses htmlparser2 instead of Cheerio and regular expressions;
  cheerio and zod are no longer dependencies.
• Tests exercise the public API against local servers; strict TypeScript.

3.0.0-beta.1

Second beta of mapsite 3.

Install: npm i mapsite@next

• Fix: works on Bun (redirects followed internally, no Dispatcher.compose) (#11)
• Fix: comma-joined Content-Type values are accepted if any is allowed (#9)
• Docs: install with mapsite@next while 3.x is in beta; v2 pins mapsite@2 (#10)

See the 3.0.0 section of the CHANGELOG for breaking changes since 2.x.

Please report issues at https://github.com/Eckhardt-D/mapsite/issues.

3.0.0-beta.0

First beta of mapsite 3: rewritten crawler, HTTP layer and parser.

Install: npm i mapsite@next

• Breaking changes and new features: see the 3.0.0 section of the CHANGELOG and the README's "Upgrading from 2.x".
• entries (lastmod, changefreq, priority, alternates) is opt-in via includeEntries.

Please report issues at https://github.com/Eckhardt-D/mapsite/issues.

v2.2.1

This is a maintenance release, in preparation for a major version release. the 2.x line will be sporadically maintained for security and bug fixes.

Changelog

• Declare Node.js 22.19.0 as the minimum supported runtime.
• Update runtime and development dependencies, remove unused dependencies, and
  commit the npm lockfile for reproducible development and CI.
• Fix TypeScript declaration generation with current Cheerio types while keeping
  the existing parsing mode and response shape.
• Give each fetch its own retry budget, including concurrent requests.
• Apply HTTP status handling and retries to authenticated requests; correctly
  decode URL credentials and support empty passwords.
• Accept missing content-type headers when strict checking is disabled and drain
  rejected response bodies.
• Apply the documented default recursion limit with partial constructor options,
  count depth per branch, and reset the depth counter for each parse operation.
• Replace fixed test ports and shared test state with independent local fixtures;
  cover nested indexes, repeated calls, concurrency, authentication, and proxies.
• Make coverage optional for local tests and check the build across Node 22, 24,
  and 26 in CI. Rebuild distributable files before packing or publishing.

v2.2.0

Changelog

• Upgrade dependencies to address security issues

Fringe Benefits

• Upgraded Undici has improved some HTTP/2 issues compared to previous version

v2.1.0

Changelog

Major parser rewrite

• Moved away from manual parsing, now using Cheerio to parse the XML
• Added support for Basic Auth urls (https://username:password@mysite.com/sitemap.xml)
• Improve performance of recursive URL collection in sitemap index files by not copying memory and mutating instead
• Improved performance by reducing the wait time between batched requests
• Improved performance by moving to Undici

v1.4.1

Changelog

• Updates default user agent
• Adds support for proxies

v1.3.2

Changelog

• Support gzip (sitemap.xml.gz) urls by checking content-type headers and automatically unzipping with zlib

v1.2.4

Changelog

• ignores Googles image:image and video:video tags

v1.1.0

Changelog

• Accepts custom userAgent option