fix(docs): stop embedding internal engineering records; close the docs corpus

[aaj3f]

Part of #1653 — the A1 corpus scoping plus the D3 cross-links. A6 (rev-stamping non-release builds) is deliberately not here; reasons at the bottom.

A1 — the embedded corpus is now a closed set

The rust-embed derive in fluree-db-docs/src/embed.rs embedded all of ../docs minus book/* and hidden dirs — which shipped docs/audit/ and docs/audit-impl/ (39 internal engineering records: branch names, worktree paths, benchmark run IDs, contributor attribution) inside every released binary, retrievable by any user or agent through fluree docs search / the MCP docs_search tool. Verified at the binary surface before and after: fluree docs search "findings register" returned four audit/… paths on origin/main and returns none on this branch; "worktree" went from four hits to zero. Worth being precise about the exposure: mdBook only builds SUMMARY.md chapters, so these records were never on the docs site — the leak was exactly the binary/agent surface, i.e. content deliberately kept off the website was being served to anyone who asked the binary.

The fix is a policy, not just an exclude: an embedded page must be linked from SUMMARY.md, listed in EMBEDDED_EXTRAS with a reason, or excluded — and four new guardrail tests keep the set closed in both directions (embedded-but-unlisted fails naming offenders; an extras entry that stops existing fails too, so the allowlist can't rot). #[include = "*.md"] also drops book.toml and scripts/llms_preproc.py, which were bytes in every binary no surface could return. The arithmetic closes exactly: 233 embedded = 231 SUMMARY-linked + 2 extras.

Disclosure: which releases shipped this. docs/audit/ first appears at tag v4.1.2 (11 records) and the set grows to 39 by v4.1.5 — so released binaries v4.1.2 through v4.1.5 all embed internal engineering records (branch names, worktree paths, benchmark run IDs, contributor attribution), retrievable via fluree docs search/docs_get and the MCP docs tools. The records were never on the docs site (mdBook builds only SUMMARY chapters); the exposure is exactly the binary/agent surface. At the CLI's default limit of 10, "findings register" returned 9/10 audit hits and "worktree" returned 8/8 — the word existed nowhere else in the corpus. This PR is labeled bug so release notes file it where someone auditing an installed binary will look.

The 44 previously-unlinked files (39 excluded + 3 promoted + 2 extras, one of which is SUMMARY.md itself) sorted as:

• 39 excluded — the audit records. They stay in the repo; the ~dozen Rust source comments citing them by repo-relative path are unaffected.
• 3 promoted into SUMMARY.md — contributing/benches.md, memory/cli/audit.md, memory/guides/hygiene-and-auditing.md. The two memory pages had inbound links from published pages that were dead on the site; contributing/benches.md's inbound links are repo-root files (BENCHMARKING.md, fluree-bench-support/README.md), so promoting it is an editorial call — the right one, but a call, not the mechanical no-judgment case.
• 2 extras — SUMMARY.md itself (the TOC; docs_tree parses it, it isn't a content page) and operations/running-fluree.md with a TODO(review): 402 lines, zero inbound links, substantial overlap with three published pages. Publishing it or folding it in is an editorial call I didn't want to make silently inside a scoping PR, so it stays searchable with the open decision visible.

Mutation-checked: a scratch page under docs/ fails the publish test naming it; removing the excludes fails three tests listing all 39 records; a bogus extras entry fails the exists test.

D3 — the glue docs

docs/cli/server-integration.md and docs/cli/load.md turned out to already be in SUMMARY.md (221d01a, which also added the docs_coverage gate for docs/cli/) — the anchor's premise was stale there, and the new corpus-wide test generalizes that same gate. What did change: docs/ai/claude-code.md now points at the fluree-cli Claude Code plugin (and the forthcoming stack-first fluree-companion), and its remote section notes that a Fluree AI stack serves its own version-pinned docs at /api/docs — shape verified against the solo source rather than taken on faith.

Why A6 isn't here

Rev-stamping would touch two different contracts and neither cheaply: this workspace has zero build scripts across 48 crates (adding one to the CLI puts a build-script dependency in every workspace compile), a baked SHA goes stale without rerun-if-changed on the resolved HEAD — and in a repo worked almost entirely through git worktrees, the obvious ../.git/HEAD is wrong in exactly the environment we use daily; a confidently wrong rev is worse than none. And the docs-stamp half isn't a field addition — fluree_db_docs::VERSION rides the wire shape of docs --json and the MCP docs_* tools, which downstream consumers parse. fluree manifest already carries the features array as the main same-version-different-surface signal, and A1 makes the docs corpus deterministic per commit. If we still want build-identity in the field, it deserves its own issue with those constraints on the table.

Tests

cargo fmt --check clean; clippy -p fluree-db-docs --all-targets clean; fluree-db-docs 19/19 (4 new); docs_coverage 5/5; fluree-db-mcp 8/8; mdbook build succeeds and the llms preprocessor stages 231 pages, matching the SUMMARY count exactly.

One pre-existing loose end left alone: bench-baselines/README.md links docs/audit/2026-06-architecture-audit.md, which no longer exists on main — unrelated to this change.

---

Review-pass remediation (post-open): parse_summary now strips HTML comment spans before scanning — a SUMMARY entry unpublished with a multi-line <!-- --> block previously kept all guardrails green while mdBook dropped the page (reproduced, then fixed; the same-line form already failed closed), and docs_tree stops listing commented-out chapters as a side effect. ./-prefixed links are normalized so a published page can't fail the guardrail with a message claiming the opposite. Three new parse tests cover both comment forms, unterminated comments, and the prefix. ai/claude-code.md no longer overstates the fluree-cli plugin's wiring (docs-only, not memory), and the index's post-#[include] filters are labeled belt-and-braces. Docs suite 22/22, coverage 5/5, mcp 8/8, fmt + clippy clean.

Note on the clippy red at HEAD: it is main's breakage, not this branch's — #1580 trips clippy::type_complexity on main itself, and every PR inherits it through the merge-ref checkout. Fix: #1658. This PR's own crates are clippy-clean under -D warnings locally.

[bplatz]

LGTM!