Sitemap discovery misses root-domain sitemap when scoring a subdirectory URL

[dacharyc]

Summary

When the scored URL is a subdirectory (e.g. https://www.swift.org/documentation/), afdocs checks for a sitemap at <base-url>/sitemap.xml — in this case https://www.swift.org/documentation/sitemap.xml. If that path returns 404, afdocs falls back to testing only the root URL and emits a single-page-sample diagnostic, even when a valid sitemap exists at https://www.swift.org/sitemap.xml.

Steps to reproduce

npx afdocs check https://www.swift.org/documentation/ --sampling deterministic --max-links 50 --format json --score

Expected: afdocs discovers and samples pages from the sitemap at https://www.swift.org/sitemap.xml.

Actual: discoverySources: ["fallback"], testedPages: 1, single-page-sample diagnostic fires.

Root cause

The discovery sequence:

1. Checks robots.txt at https://www.swift.org/robots.txt — found, but no Sitemap: directive
2. Tries https://www.swift.org/documentation/sitemap.xml — 404
3. Falls back to testing only the root URL

Step 2 is path-scoped to the base URL. It never tries https://www.swift.org/sitemap.xml — the conventional root-domain location.

Expected behavior

When the base URL is a subdirectory and the path-relative sitemap returns 404, fall back to checking <scheme>://<host>/sitemap.xml (and <host>/sitemap-index.xml variants) before giving up on sitemap discovery.

Sitemaps are almost never placed under a subdirectory path — they're nearly always at the root. The scoped path check has low hit rate, while the root-domain fallback would recover a significant fraction of cases like this one.

Notes

• https://www.swift.org/sitemap.xml returns HTTP 200 with 446 URLs, 47 of which are under /documentation/
• https://www.swift.org/documentation/sitemap.xml returns HTTP 404
• The robots.txt (User-agent: *, Disallow: /builds/) has no Sitemap: line

[dacharyc]

Investigated — the title is slightly misleading. PR #82's sitemap discovery candidate fix is already in place and is correctly fetching https://www.swift.org/sitemap.xml. The actual cause for the swift.org failure is downstream: every URL inside that sitemap is published on the bare host (https://swift.org/...), and the origin filters in shouldInclude() (src/helpers/get-page-urls.ts:743) and scopeUrls() (src/checks/observability/llms-txt-coverage.ts:300) both used strict origin !== comparison. So all 446 URLs got dropped, the sitemap result was empty, and afdocs fell back to single-page sampling — which is what triggered single-page-sample and zeroed the score.

Same root cause is also why the url-stability category was at risk of seeing these as cross-host redirects (though isCrossHostRedirect() already handled www-equivalence; the bug was only in the sitemap filter path).

Fix in #84: adds isSameOriginIgnoringWww() (built on the existing isWwwVariant helper) and uses it in both filter sites. Tests cover both directions of www mismatch plus a regression test confirming truly cross-host URLs are still rejected.