feat(sitemap): add support for "lastmod"

[slorber]

Motivation

Fix #2604

Add new sitemap options:

{
  lastmod: null | 'date' | 'datetime'
  priority: null,
  changefreq: null,
}

• Add a new sitemap option lastmod that permits to output <lastmod> tags in the sitemap, formatted as YYYY-MM-DD date, or ISO datetime
• Allow to disable useless <changefreq> and <priority> sitemap tags (Google doesn't use them) with null values

Important notes:

• lastmod only works for routes backed and requires plugin authors to add new route metadata when calling addRoute()
• All our official content plugins provide this route metadata
• If plugins allow to provide a last updated time as front matter, then it's possible for the sitemap plugin to use this explicit date in priority. Otherwise, a timestamp will be inferred from the Git history of the content source file.
• We are able to output a <lastmod> tag for React page components as well. However it will only use the React page source Git history, and will not check history of its dependencies.
• Not all routes will have a <lastmod> tag. If a third-party plugin does not provide the new metadata, the sitemap can't infer a proper value. According to Google recommendations, it's only relevant to use <lastmod> tags for routes backed by content, and not routes aggregating multiple documents (tags, blog pagination, blog archive, doc autogenerated index...)

Test Plan

Unit tests + dogfood on our website

Test links

• Website sitemap: https://deploy-preview-9954--docusaurus-2.netlify.app/sitemap.xml
• Sitemap options doc: https://deploy-preview-9954--docusaurus-2.netlify.app/docs/api/plugins/@docusaurus/plugin-sitemap/
• RouteMetadata doc: https://deploy-preview-9954--docusaurus-2.netlify.app/docs/api/plugin-methods/lifecycle-apis/#addRoute

Related issues/PRs

Supersed PR #9234

[netlify]

✅ [V2]

Name	Link
🔨 Latest commit	d92d80d
🔍 Latest deploy log	https://app.netlify.com/sites/docusaurus-2/deploys/65fabb7a60eb5c00078573c5
😎 Deploy Preview	https://deploy-preview-9954--docusaurus-2.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[github-actions]

⚡️ Lighthouse report for the deploy preview of this PR

URL	Performance	Accessibility	Best Practices	SEO	PWA	Report
/	🔴 46	🟢 98	🟢 96	🟢 100	🟠 88	Report
/docs/installation	🟠 59	🟢 96	🟢 100	🟢 100	🟠 88	Report
/docs/category/getting-started	🟠 76	🟢 100	🟢 100	🟢 90	🟠 88	Report
/blog	🟠 69	🟢 100	🟢 100	🟢 90	🟠 88	Report
/blog/preparing-your-site-for-docusaurus-v3	🟠 59	🟢 96	🟢 100	🟢 100	🟠 88	Report
/blog/tags/release	🟠 71	🟢 100	🟢 100	🟠 80	🟠 88	Report
/blog/tags	🟠 76	🟢 100	🟢 100	🟢 90	🟠 88	Report

[github-actions]

Size Change: +45 B (0%)

Total Size: 992 kB

Filename	Size	Change
website/build/assets/js/main.********.js	765 kB	+45 B (0%)

ℹ️ View Unchanged

Filename	Size
website/.docusaurus/globalData.json	75.4 kB
website/build/assets/css/styles.********.css	114 kB
website/build/index.html	37.9 kB

_{compressed-size-action}

[johnnyreilly]

This is excellent work; I like that you're looking to make the default sitemap include lastmod by default with Docusaurus v4

[slorber]

LGTM 🎉

Thanks for the review 🙏

[razzeee]

Is it expected, that lastmod is always today? (might be date of last deploy)
For every page?

See https://docs.flathub.org/sitemap.xml
Code is here https://github.com/flathub-infra/documentation

[slorber]

@razzeee you need to configure your CI to fetch the full git history.
I may have missed to mention that in the plugin docs, but it should be there in docs/blog docs.

[johnnyreilly]

You may need:

      - name: Checkout 📥
        uses: actions/checkout@v4
        with:
          # Number of commits to fetch. 0 indicates all history for all branches and tags.
          fetch-depth: 0
          submodules: true

https://github.com/johnnyreilly/blog.johnnyreilly.com/blob/2a4665f3accd44f9199b8232310368dedac37f19/.github/workflows/build-and-deploy-static-web-app.yml#L33C1-L38C27

[razzeee]

Thanks, working now

[pmarschik]

I've now got around to trying the released version.
On our CI it works as expected, however locally I have the git config log.showSignature = true which causes npm docusaurus build to fail with:

[ERROR] Loading of version failed for version current

[ERROR] Error: Unable to build website for locale en.

[...]

Error: Can't process doc metadata for doc at path path=/Users/<redacted> in version name=current

[cause]: Error: An error occurred when trying to get the last update date
    at getGitLastUpdate (/Users/<redacted>/node_modules/.pnpm/@docusaurus+utils@3.2.0_@docusaurus+types@3.2.0_@swc+core@1.4.11_esbuild@0.17.19/node_modules/@docusaurus/utils/lib/lastUpdateUtils.js:43:19)
        
[cause]: Error: Failed to retrieve the git history for file "/Users/<redacted>/file.mdx" with unexpected output: Good "git" signature for somebody@example.org with ED25519-SK key SHA256:<somekey>
  <sometimestamp>,<some name>
      at getFileCommitDate (/Users/<redacted>/node_modules/.pnpm/@docusaurus+utils@3.2.0_@docusaurus+types@3.2.0_@swc+core@1.4.11_esbuild@0.17.19/node_modules/@docusaurus/utils/lib/gitUtils.js:63:15)

To make the logic in packages/docusaurus-utils/src/gitUtils.ts more robust we could:

• Set -c log.showSignature=false on the CLI. But this needs to be adapted for any new setting that modifies the output.
• Mark the output that we want to parse: set --format=COMMIT-TIMESTAMP:%ct${includeAuthor ? ',COMMIT-AUTHOR:%an' : ''} and extend the regex with the tags and allow multiline matches.

[slorber]

Thanks @pmarschik , I adopted both solutions at once in #10022

and extend the regex with the tags and allow multiline matches.

Not sure what you mean here

[pmarschik]

Thanks @pmarschik , I adopted both solutions at once in #10022

and extend the regex with the tags and allow multiline matches.

Not sure what you mean here

@slorber Thanks for the quick fix.

I meant exactly what you merged in #10022. Except with multiline I meant the RegExp could be /^RESULT:(?<timestamp>\d+),(?<author>.+)$/m instead of /(?:^|\n)RESULT:(?<timestamp>\d+),(?<author>.+)(?:$|\n)/

[slorber]

ah I see thanks, just a different way to express the same thing I guess (and yours look better 😅 )