📝 docs: nest nav, add tutorials & workflows, generate llms.txt

[gaborbernat]

The documentation had grown to a flat left-nav of 39 how-to guides, 24 explanation pages, and 65 migration guides with no grouping, a hand-maintained llms.txt pair that already listed only half the how-tos, and a tutorial track that stopped at the string helpers while the real draw — scrape a page, pull the article, emit Markdown — had no learning path. 📚 This overhaul reorganizes the navigation, adds the tutorials and workflow recipes the competitor-usage mining surfaced, and makes the AI map and source links generate themselves so they cannot drift.

The left nav now groups every index by the same eight namespaces the reference already used (parse/DOM, detect, query, clean, convert, extract, build, serialize), a mechanical split of the existing flat toctrees with no page moved. A set of Sphinx extensions carries the polish: sphinx-design for the landing-page card grid, sphinx-sitemap, sphinxext-opengraph, sphinx-notfound-page, sphinx-last-updated-by-git, sphinx-codeautolink to link names in the runnable examples to the reference, and the builtin linkcode wired to GitHub source. A build-time extension replaces the two static llms.txt files: it walks the toctree, reads each page's title and first paragraph, and writes a curated llms.txt and a full llms-full.txt into the HTML root, so the map stays in step with the pages.

On top of the structure, the tutorial track grows from five pages to eleven with real-world walkthroughs, new workflow how-tos for the gaps the mining found (scrape-for-LLM, migrating off HTMLParser, feeds, page metadata, truncating), and a task-shaped troubleshooting page for the gotchas that trip newcomers. The README is refreshed to the current surface — XSLT, CSSOM, validation, streaming rewrite, forms, standalone detect, HTML+CSS minify, and the CLI — with a second quickstart for the parse→prune→Markdown workflow.

Every code block in the tutorials and how-tos runs under sphinx-build -W + doctest, and a new sphinx-lint pre-commit hook and docs-linkcheck tox env guard the prose. A final pass runs the whole docs/ tree through the house no-slop rules.

[codspeed-hq]

Merging this PR will not alter performance

✅ 91 untouched benchmarks
⏩ 18 skipped benchmarks¹

---

_{Comparing gaborbernat:docs/overhaul (301270b) with main (4e6b1ce)}

Footnotes

1. 18 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩