fix(site-docs): return HTTP 404 for unknown routes (stop soft-404)

[zxch3n]

Problem

Unknown URLs on https://lody.ai were soft-404s: HTTP 200, homepage title/description/OG, and <link rel="canonical" href="https://lody.ai/">. Crawlers treated junk paths as duplicate homepage.

Production repro (before):

curl -sI https://lody.ai/this-path-should-not-exist-seo-test-9f3a
# HTTP/2 200
# <title>Lody — Your Local, Private Coding Agent</title>
# <link rel="canonical" href="https://lody.ai/" />

Same for unknown /docs/... paths.

Root cause

site-docs is static (TanStack Start prerender → out/client on Cloudflare Pages). There was no root 404.html. Cloudflare then treats the site as an SPA and serves index.html with 200. Dev/SSR already returned 404; production static hosting did not.

/app is a separate product HTML on the same host (<title>Lody</title>). A catch-all _redirects /* /index.html 200 would steal it. This PR does not add that.

Fix

1. Dedicated 404 UI + head (SiteNotFound, /404, /zh/404): title Page not found | Lody / 页面不存在 | Lody; noindex, follow; no canonical and no og:url.
2. Prerender /404 → 404.html (autoSubfolderIndex: false).
3. Strip SPA hydration from 404.html after prerender (scripts/finalize-404-html.mjs). Hydrating the app on a junk URL blanked the page / showed "Something went wrong". Static 404 is now a complete HTML document.
4. Static host (scripts/static-host.mjs + preview:static): missing files → HTTP 404 + 404.html. Pretty /404 also 404 (error document, not a 200 page).
5. Docs/blog/changelog splat head() uses notFoundHead instead of index/homepage head.
6. Sitemap excludes /404, /zh/404, /home, /zh/home.

SEO choice: static 404.html cannot bake a per-request canonical. noindex, follow + no canonical is correct. Homepage canonical was the bug.

Local commands

corepack pnpm install
corepack pnpm --filter @lody/site-docs test
corepack pnpm --filter @lody/site-docs typecheck
corepack pnpm --filter @lody/site-docs dev          # :3002
corepack pnpm --filter @lody/site-docs build        # prerender + finalize-404-html
corepack pnpm --filter @lody/site-docs preview:static  # :4173, CF-equivalent

Do not add Vite configurePreviewServer 404 middleware — TanStack prerender uses vite.preview() to fetch pages before they exist on disk.

Curl evidence (preview:static :4173)

/this-path-should-not-exist-seo-test-9f3a  → 404  title=Page not found | Lody  robots=noindex, follow  canonical=NONE
/docs/this-doc-does-not-exist-seo-test     → 404
/docs/core-concepts/nope                   → 404
/%E4%B8%8D%E5%AD%98%E5%9C%A8               → 404
/                                          → 200  title=Lody — Your Local...  robots=index, follow  canonical=https://lody.ai/
/docs/                                     → 200
/blog/introducing-lody                     → 200
/price                                     → 200
/zh/                                       → 200
/zh/docs/                                  → 200
/docs/?utm=seo                             → 200

Adversarial review

Case	Result
Trailing slashes on known routes	200 via index.html
Nested unknown /docs/.../nope	404
Encoded unknown path	404
Query strings on known routes	200
i18n /zh/ + zh docs	200
Accidental noindex on real pages	Homepage index, follow
Pretty /404	HTTP 404 (error document)
Hydration on junk URL	Fixed by stripping app JS from 404.html
/app	Not in this tree; local 404. Production is a separate worker — no catch-all _redirects

Residual risks

• Cloudflare dashboard SPA fallback (/* /index.html 200) could still win over 404.html. Confirm Pages settings after deploy.
• One English 404.html for all unknown URLs (locale-specific first paint is a follow-up).
• vite dev 404 title is still root "Lody Docs" (HTTP 404, no homepage canonical). Crawlers hit static 404.html.
• Prerender of /404 itself is HTTP 200 (failOnError requires res.ok).

Walkthrough

Unknown URL shows dedicated Not Found page
Document title is Page not found | Lody
Homepage still 200 after visiting 404
Unknown docs path is Not Found
Real docs introduction still 200
Chinese homepage still 200

soft_404_fixed_unknown_url_and_known_pages.mp4

_{To show artifacts inline, enable in settings.}