From 214a9c0444750db5b5b322a2925148b01eb1dbec Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sun, 17 May 2026 17:57:54 +0000 Subject: [PATCH 1/2] Initial plan From 3eac44191d1c63bf81737292ef0c852ed8810df8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sun, 17 May 2026 18:10:02 +0000 Subject: [PATCH 2/2] fix(docs): align robots sitemap URL and document project-site discovery paths Agent-Logs-Url: https://github.com/github/gh-aw/sessions/e6d6645f-670b-4922-9329-5236bcaf7129 Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com> --- docs/README.md | 10 ++++++++++ docs/public/robots.txt | 2 +- docs/public/sitemap.xml | 6 ++++++ 3 files changed, 17 insertions(+), 1 deletion(-) create mode 100644 docs/public/sitemap.xml diff --git a/docs/README.md b/docs/README.md index 773dcdef519..e8c464e14bd 100644 --- a/docs/README.md +++ b/docs/README.md @@ -44,6 +44,16 @@ The sitemap (`/gh-aw/sitemap-index.xml`) is **only generated during a production If a CI pipeline or automated tool checks for the sitemap URL during a local preview, it will receive a 404 response. To verify the sitemap, run `npm run build` followed by `npm run preview`. +### Robots/AI discovery paths on GitHub Pages project sites + +This docs site is deployed as a GitHub Pages **project site** under `/gh-aw/` (see `base: '/gh-aw/'` in `astro.config.mjs`). + +- `robots.txt` is served at `/gh-aw/robots.txt` +- AI discovery file is served at `/gh-aw/.well-known/ai.txt` +- AI metadata files are served under `/gh-aw/ai/` + +Root-level endpoints on `https://github.github.com/` (for example `/robots.txt`) are controlled by the main `github.github.com` site, not this repository. + ## Want to learn more? Check out [Starlight’s docs](https://starlight.astro.build/), read [the Astro documentation](https://docs.astro.build), or jump into the [Astro Discord server](https://astro.build/chat). diff --git a/docs/public/robots.txt b/docs/public/robots.txt index 618266d1429..22b25d4ceba 100644 --- a/docs/public/robots.txt +++ b/docs/public/robots.txt @@ -86,4 +86,4 @@ Allow: / User-agent: Diffbot Allow: / -Sitemap: https://github.github.com/gh-aw/sitemap-index.xml +Sitemap: https://github.github.com/gh-aw/sitemap.xml diff --git a/docs/public/sitemap.xml b/docs/public/sitemap.xml new file mode 100644 index 00000000000..4c4754adf34 --- /dev/null +++ b/docs/public/sitemap.xml @@ -0,0 +1,6 @@ + + + + https://github.github.com/gh-aw/sitemap-index.xml + +