Skip to content

v3.0.0

Latest
Latest

Choose a tag to compare

View all tags
@Eckhardt-D Eckhardt-D released this 14 Oct 05:41
· 1 commit to master since this release
v3.0.0
ad06bc2

The first stable 3.x release, now latest on npm. The 2.x line stays maintained on the v2 branch for bug and security fixes (npm install mapsite@2).

npm install mapsite

Changelog

The constructor, option names, run, fromBuffer and the { type, urls, errors }
result are unchanged. The internals were rewritten; see "Upgrading from 2.x" in the
README for the full list of differences.

Fixed

  • Works on Bun: redirects are followed internally instead of via Dispatcher.compose.
  • proxy works on Bun, whose built-in undici ignored ProxyAgent and sent
    requests directly; proxied requests now use Bun's fetch proxy option.
  • A Content-Type header with several comma-joined values is accepted if any is allowed.

Known limitations

  • On Bun, timeout is not enforced and close() does not close connections:
    Bun's built-in undici ignores them.

Breaking

  • type reports the root document: a sitemap index is "index" (2.x returned
    whichever document finished last).
  • Removed the debug option, SitemapParser.getLinesFromText, and the
    currentDepth, currentRetry and maximumDepth properties.
  • maximumRetries defaults to 1 regardless of other options (2.x used 3 only
    without an options object) and accepts 0. Only transient failures are retried,
    with exponential backoff; 4xx responses other than 408, 425 and 429 are not.
  • Content types are matched exactly instead of by substring.
  • Invalid options throw a TypeError naming the option instead of a ZodError.
  • Pages and sitemaps seen more than once are skipped; pages come back in document
    order.
  • Documents over 50 MiB (compressed or decompressed) are rejected; see
    maximumResponseSize.
  • <loc> is read only as a child of <url> or <sitemap>.

Added

  • entries on the result, behind the includeEntries option (default false): lastmod, changefreq, priority and hreflang
    alternates next to each URL.
  • discover(site) finds sitemaps through robots.txt (falling back to
    /sitemap.xml).
  • stream(url) yields entries as they are found, and stops fetching when you stop
    iterating.
  • AbortSignal support on every method, and close() / await using.
  • Plain-text sitemaps, RSS and Atom feeds.
  • Redirects are followed, and gzip is detected from the data rather than the
    content type.
  • Options concurrency, maximumResponseSize and retryDelay.
  • proxyMode: 'fallback': fetch directly and only use the proxy for sites that
    answer 401, 403 or 429.
  • blockPrivateAddresses: refuse direct connections to non-public addresses,
    including through redirects and DNS, for crawling user-supplied URLs.
  • ESM and CommonJS builds with an exports map.

Changed

  • Rewrote the crawler and HTTP layer; sitemaps are fetched concurrently without the
    fixed three-at-a-time batches and delays.
  • Parsing uses htmlparser2 instead of Cheerio and regular expressions;
    cheerio and zod are no longer dependencies.
  • Tests exercise the public API against local servers; strict TypeScript.
Assets 2
Loading