docs(site): make the llms.txt artifacts discoverable, and fix the landing-card parser

[bloxster]

Why

docs.erigon.tech/llms.txt and /llms-full.txt are published, but nothing points anything at them. They are absent from robots.txt, absent from sitemap.xml, and unlinked from every built HTML page. They are static files under static/, so Docusaurus never routes them and the default sitemap omits them.

The effect is measurable. Asked a single Erigon flag question, ChatGPT read 60 sources — roughly 8 doc pages, ~20 GitHub issue threads, 6 unrelated MCP projects, and Wikipedia's article on HTTP — and touched neither file. We are paying to generate a corpus nothing can find.

What this does

Commit 1 — discoverability.

• Two <link rel="alternate" type="text/plain"> head tags on every page advertising both .txt URLs.
• createSitemapItems appends both URLs to the sitemap, preserving every default entry via defaultCreateSitemapItems(rest).
• A reader-facing section on the MCP page — where someone already asking "how do I point a model at Erigon" will be — plus a one-line pointer from why-using-erigon.

robots.txt is deliberately untouched: no standard directive advertises llms.txt, and the Sitemap: line already there now leads to both files.

The prose lands on mcp.mdx rather than why-using-erigon.mdx because the latter is a card-grid landing page whose body synthesize_landing() replaces wholesale — prose added there would render on the site but never reach llms-full.txt.

Commit 2 — a live corpus bug found while writing the above.

_LANDING_CARD_RE matched a whole card with one pattern: [^<]+ for the title and description text, (?:.*?) for the gaps, under re.DOTALL. [^<]+ cannot cross a <, so a description containing inline markup (<strong>, <code>) fails to match where it stands — and the engine then scans forward through the permissive gap and matches the next card's description and </Link>, swallowing the card in between and pairing a title with the wrong description.

This is live today, on the page whose job is explaining what makes Erigon different. why-using-erigon has 11 cards; llms-full.txt carried 8:

Card	What the corpus said
Immutable, Decentralised Data	carried Staged Sync's description
Flexible Pruning	carried RPC Providers' description
Staged Sync, RPC Providers & Large Stakers, Developers	absent entirely

Fixed by parsing in two stages — match each <Link> block first, then find title and description within that block only, which makes a card boundary unrepresentable.

Plus a count guard. This failed silently for as long as it existed because --check only compares generated output against committed output, which makes a systematic generator bug invariant under it: CI stays green while the corpus is wrong. The guard compares parsed cards against lp-card-title occurrences and fails loudly on a mismatch.

A sweep of all 8 card-grid pages confirms why-using-erigon.mdx is the only file affected.

Verification

• npm ci && npm run build clean.
• Sitemap: 75 URLs including both .txt; /search still excluded; 73 per-page lastmod preserved.
• Head tags present on 198/199 pages (the miss is a redirect stub).
• generate-llms.py --check green; 81 tests pass.
• The three regression tests were each confirmed to fail against the old parser and pass against the new one.

Note on branches

docs-deploy.yml currently exists only on release/3.5, so this publishes once the 3.6 deploy switchover lands.

[bloxster]

Companion PR for main: #23336 — same two commits, so the change is inherited by the next release branch cut from main rather than needing a re-port at cutover.