feat(utils): add sitemapFilter option to parseSitemap

[sbruinsje]

Motivation

When working with sitemap index files, parseSitemap currently follows all child sitemaps unconditionally. Sometimes sitemap indexes contain hundreds of child sitemaps, for instance, a child sitemap for every month going back 15 years (e.g., /articles-2010-01.xml through /articles-2026-03.xml). If you're only interested in the last 2 years of content, there's no way to skip the irrelevant ones without fetching and parsing all of them.

This PR adds a sitemapFilter callback option that lets you control which child sitemaps to skip, based on their URL.

Changes

• Add optional sitemapFilter?: (sitemapUrl: string) => boolean to ParseSitemapOptions
• When provided, each child sitemap URL discovered in a sitemap index is passed through the filter before being fetched. Return true to include, false to skip.
• When not provided, behavior is unchanged all nested sitemaps are followed.
• Skipped sitemaps are logged at debug level.

Example usage

// Only follow child sitemaps from the last 2 years
for await (const url of parseSitemap(sources, undefined, {
    sitemapFilter: (url) => /articles-202[4-5]/.test(url),
})) {
    // ...
}

[janbuchar]

I haven't read the code yet, but do I understand it correctly that this new callback is invoked for sitemapindex urls? And that it's not used for actual page urls?

[sbruinsje]

Yes that is correct. Only for child sitemap urls in a sitemap index.

[janbuchar]

LGTM, but let's think about the naming.

[janbuchar]

Let's think about alternative names for this option. How about nestedSitemapFilter?

[sbruinsje]

Yes that makes sense. A nested sitemap filter is exactly what it is.

[janbuchar]

Cool, let's go with that then