feat(docs): SEO and AEO optimization

[imran-siddique]

What and why

Makes the TRACE docs discoverable by both search engines and AI answer engines (LLMs). Today the site has no llms.txt, no structured data, and no robots.txt, so it is under-picked-up by Google AI Overviews, Perplexity, ChatGPT, and Claude. This PR is additive scaffolding plus answer-first framing on non-normative pages.

Changes

AEO (LLM / answer-engine pickup)

• llms.txt and llms-full.txt generated at the site root via the mkdocs-llmstxt plugin, with a curated summary and sectioned links (Getting started, Specification, Integration, Platforms).
• robots.txt at the site root that welcomes major AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot) and points to the sitemap.
• FAQPage JSON-LD plus a visible answer-first FAQ on the home page.

SEO

• Organization, WebSite, and SoftwareApplication JSON-LD plus OpenGraph and Twitter card meta on every page, injected through a new theme override (overrides/main.html). Canonical is left to Material (it already emits it because site_url is set).
• Per-page description front matter and a TL;DR block on the home, quickstart, and trust-levels pages.
• sitemap.xml continues to auto-generate (28 URLs).

Build

• Added mkdocs-llmstxt to requirements-docs.txt.
• The docs CI assemble step now copies robots.txt into the build root (the site uses docs_dir: . with a .docs_build assemble), and the workflow rebuilds on overrides/** and robots.txt changes.

Verification checklist (built locally via the CI-replicated assemble)

• mkdocs build succeeds (non-strict; see caveat).
• site/llms.txt (1.5 KB) and site/llms-full.txt (93 KB) present at root.
• site/robots.txt served at root with sitemap reference.
• site/sitemap.xml present, 28 URLs incl. /.
• Home page has 2 JSON-LD blocks, both validate: [Organization, WebSite, SoftwareApplication] and FAQPage.
• OG and Twitter meta present on pages.
• No normative spec text changed.

Caveats / follow-ups

• mkdocs build --strict still aborts, but only on 7 pre-existing warnings unrelated to this PR (broken links in README.md and GOVERNANCE.md, pages not in nav, docs/tutorials/agt-adapter.md links). Worth a separate cleanup PR to enable strict.
• No og:image yet: the repo only has icon.svg (no raster social card). Adding a 1200x630 PNG is a good follow-up. The Material auto social-card plugin was intentionally not enabled because the docs CI lacks the cairo/pango system libraries and would break the build.
• The visible FAQ and answer-first content sit on the nav "Home" page (/docs/). The true site root / is README.md; it still gets the site-wide JSON-LD and OG meta via the override. Aligning the site root with the docs home is a separate structural decision.

Do NOT merge without a Maintain+ review (the maintainer-approval gate applies).