DOC-6939 Stop publishing HTML-commented content to the AI outputs

[andy-stark-redis]

Stops content inside HTML comments reaching the JSON feed and the per-page Markdown — item D4 of DOC-6939.

The defect

Hugo runs with unsafe = true, so an HTML comment passes straight through to the rendered page. It's invisible to readers, which is exactly how authors park prose that shouldn't be published yet. Nothing in the AI output path knew that, so commented text was being treated as publishable content.

We were publishing to AI consumers exactly what we withhold from readers. That's worse than a formatting defect — an assistant can cite a feature as documented when we have deliberately not documented it.

The case that surfaced it is develop/clients/observability, where a comment holds an entire "Tracing overview" section, kept because the explanation is good but the clients don't support tracing yet:

Artifact	Before	After
Rendered page	Hidden inside an HTML comment ✓	unchanged ✓
sections[] in the feed	A real section, 1,610 chars of prose	gone
Per-page Markdown	Present	gone
Metadata TOC	Present, pointing at a non-existent anchor	gone

Across the corpus, 56 files carry HTML comments totalling about 22,000 characters. Only 2 of those comments contain a heading, which is why just one showed up in an anchor-parity check — the other 54 leak their prose into whatever section encloses them, invisible to any id-based metric.

The fix, and why it's line-anchored

A line-anchored strip in process-markdown-content.html (the choke point feeding both the JSON content field and the Markdown output) and in toc-from-markdown.html (which reads .RawContent directly).

The open delimiter must be at the start of a line, and that's load-bearing. HTML comments legitimately appear inside code examples — the Lettuce pages carry a Maven snippet with </version> <!-- Check for the latest version on Maven Central --> mid-line, and stripping that damages a sample a reader sees. Column-0 is what separates an author's parked block from a comment inside a code sample:

Pattern	Removes	Matches inside a fenced code block
Unanchored (?s)<!--.*?-->	214 comments / 24,560 chars	9
Line-anchored (?ms)^<!--.*?-->	48 comments / 19,232 chars	0

Both the literal and entity-escaped forms are matched, because .RawContent arrives escaped in some contexts.

Verification

Built before and after, comparing every page's content field:

• 5,706 pages byte-identical
• 27 shrank, and every one of the 27 is explained by having a line-anchored comment
• 0 grew, 0 changed pages unexplained
• Zero line-anchored comments survive into sections[] or the .md output
• Duplicate section ids 0, duplicate example ids 0, content_hash verifies 5,687/5,687

What a reviewer should focus on

• The rendered site is untouched. process-markdown-content.html feeds only the JSON and Markdown output formats, not the HTML, so nothing readers see changes.
• 27 pages lose content from their AI outputs. That's the intent, but it is content removal, so the byte-diff above is the thing to trust.
• If the anchoring is ever relaxed, it will start eating comments out of code samples at 9 known sites.

Deliberately not fixed in the content

The comment on observability.md is a legitimate authoring choice — good prose held back until the clients catch up. Removing it would have fixed one page and left 55. The defect is that our pipeline treated invisible content as publishable.

Not in this PR

• Inline comments, about 2,800 characters, still reach the outputs. Almost all of it is part of a code sample and belongs there.
• A block comment written at column 0 inside a fenced code block would still be stripped. None exists today.

Relationship to #3757

Independent. #3757 (D3 + C7) changes how titles and ids are derived; this changes what content is considered publishable. Both edit toc-from-markdown.html but in different regions — line ~27 here against lines 47+ there — so they merge cleanly in either order.

🤖 Generated with Claude Code

---

Note

Medium Risk
Intentional content removal from JSON/Markdown for ~27 pages; wrong regex anchoring could damage code samples, but rendered site HTML is out of scope.

Overview
Stops author-parked HTML comment blocks from being treated as publishable content in the JSON feed, per-page Markdown, and metadata TOC—while leaving the normal Hugo HTML site unchanged.

process-markdown-content.html now removes line-anchored <!-- ... --> blocks (and entity-escaped equivalents) before embeds, shortcodes, and other transforms, so sections[] / body text no longer include prose that readers never see (e.g. a commented-out “Tracing overview” on observability). The pattern requires the opening delimiter at column 0 so mid-line comments inside code samples (Maven snippets, etc.) are not stripped.

toc-from-markdown.html applies the same line-anchored strip after fenced code blocks are removed, so commented headings do not appear as TOC entries with dead anchors.

Scope: ~27 AI-facing pages shrink by design; inline comments and rendered HTML are unaffected. Relaxing line anchoring would risk stripping comments inside fenced code at known sites.

^{Reviewed by Cursor Bugbot for commit 2c4e608. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

DOC-6939

[github-actions]

🧠 Redis Memory

Found 5 related items from repository history:

• Commit —

• Commit —

• Commit —

• Commit —

• Commit —

Memory updated at 2c4e608

[dwdougherty]

LGTM.