Skip to content

Locale detection misses multi-locale sites when localized URLs are under 50% of the sitemap #98

Description

@brett0000FF

What happened?

Running the root check against https://docs.datadoghq.com reports llms.txt coverage of approximately 7,465 / 24,364 pages.

Datadog's default English documentation is unprefixed. Translations use these prefixes:

  • /fr/**
  • /ja/**
  • /ko/**
  • /es/**

The sitemap contains approximately 11,061 localized URLs out of 26,513 total URLs (41.7%).

detectLocalePosition() only recognizes multiple locale codes when they collectively cover more than 50% of the URLs. Because Datadog's translated pages are below that threshold, locale detection returns null and the translated pages remain in the llms-txt-coverage denominator.

The single-locale structural-duplication fallback does not apply because Datadog has four detected locale codes.

Product-scoped checks such as https://docs.datadoghq.com/infrastructure pass because base-path scoping naturally removes the locale-prefixed copies.

Public result: https://buildwithfern.com/agent-score/company/datadog-infrastructure-and-application-monitoring#checks

What did you expect?

The coverage check should recognize the unprefixed pages as the default locale and exclude recognized locale-prefixed translations from the coverage denominator, even when translations comprise less than 50% of the sitemap.

Alternatively, an explicitly configured preferred/default locale should be honored by llms-txt-coverage.

How to reproduce

Using afdocs v0.18.7:

npx afdocs@0.18.7 check https://docs.datadoghq.com \
  --format json \
  --score

Environment

  • afdocs version: v0.18.7
  • Node.js version: 24.x
  • OS: macOS Tahoe 26.5.1

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions