DOC-6939 Render metadata TOC titles with Hugo's renderer - #3757
Merged
Conversation
The metadata table of contents stripped inline code markers first and then treated
any remaining asterisk or underscore as emphasis, so by that point nothing could
tell that a character had come from inside a code span. 841 entries across 263
pages lost characters, publishing incorrect API signatures on the redisvl
reference pages -- "**kwargs" arrived as "*kwargs" and "redis_url" as "redisurl".
Title flattening now goes through RenderString and plainify, so Goldmark's actual
CommonMark rules apply, including the ban on intraword underscore emphasis, which
is why snake_case now survives whether or not it sits in a code span. No regex
reproduces that rule, which is the reason for handing the job over rather than
patching the patterns.
Two things about the Hugo side are worth knowing before touching this. markdownify
renders the string as a block, so a heading beginning "1. " parses as an ordered
list and loses its number -- 299 headings start that way, and Hugo's anchor
includes the number, so dropping it broke the deep link as well as the title.
Switching to RenderString with display "inline" made no difference at all: the
results were byte-identical, so "inline" does not stop block parsing. Escaping the
marker before rendering does.
The explicit-anchor extraction also turned out to be in the wrong place. It ran
after the title cleaning, so the old regexes ate the underscores out of anchors
such as the one on redis.error_reply, yielding an id of redis.errorreply and a
dead deep link on eight lua-api entries. It now lifts the anchor off the raw
heading before anything touches it, which is also what the TypeScript side does.
Two headings on the RDI Aurora page used a raw HTML anchor tag rather than Hugo's
brace syntax, which the renderer correctly strips, leaving them as the only two
ids that got worse. Converting them to brace syntax was cheaper than teaching the
partial a second anchor form for two headings, and the anchor value is unchanged so
existing links still resolve.
Mangled entries drop from 841 to zero. Titles change on 914 entries, 858 of them
regaining characters. Of the 20 ids that change, all 20 now match the anchor Hugo
renders and none regresses, taking TOC id parity from 94.46% to 94.51% measured
over all 37,566 entries rather than a sample. Build time is unchanged at about two
and a quarter minutes, so rendering 41,000 headings costs nothing measurable.
One measurement caveat for anyone re-running this: the last apparently-mangled
entry is not a defect. A write-behind heading escapes its own backticks, so they
are literal characters and the rendered title keeps them, matching the HTML.
Learned: RenderString with display "inline" does not stop Goldmark parsing block structures -- a heading beginning "1. " still becomes an ordered list and loses its number, byte-identically to markdownify, so the marker has to be escaped before rendering
Constraint: keep the leading ordered-list marker escaped before rendering, or 299 headings lose their number and stop matching the anchor Hugo generates for them
Constraint: lift the explicit {#anchor} off the raw heading before any rendering or cleaning touches it, or whatever runs first gets to mangle the anchor and the deep link dies silently
Rejected: markdownify for title flattening | it renders the string as a block, so leading list markers and blockquote markers are consumed
Rejected: skipping emphasis stripping when the heading contains a backtick | it leaves snake_case mangled outside code spans and reimplements CommonMark's intraword rule by hand
Gaps: sections[].title still holds raw Markdown, so it no longer string-matches the rendered TOC title, and section-id anchor parity is untouched at 91.9% pending the slug alignment in C7
Ticket: DOC-6939
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Contributor
🧠 Redis MemoryFound 5 related items from repository history: Memory updated at c2c9c0b |
Contributor
Section ids now match the anchor Hugo renders on the page, so url#section-id deep
links resolve. Parity goes from 79.6% before this branch, through 91.9% once
explicit anchors were parsed, to 100.0% over the 2,594 sections in a 400-page
sample. The metadata table of contents goes from 94.51% to 99.995% over all
37,566 entries.
Hugo has no autoHeadingID setting, so it uses Goldmark's default github style.
Five rules make it up and all five were wrong before. Underscores are kept, so
redis_url stays redis_url instead of collapsing to redis-url. Runs of hyphens are
kept. Whitespace runs are not collapsed, so "The special $ ID" anchors as
the-special--id, the discarded dollar leaving the spaces either side of it.
Leading and trailing hyphens are not trimmed: "Negation !" really does anchor as
"negation-" and "- and + special IDs" as "--and--special-ids". And only ASCII
whitespace and decimal digits count, so a non-breaking space is discarded rather
than hyphenated, and the subscripts in "Naming convention: LVQ<B1>x<B2>" vanish.
Worth knowing how those rules were established, because it was not all from the
same evidence. Harvesting 3,616 heading-and-anchor pairs out of the rendered HTML
took the algorithm from 80.2% to 90.3% and showed that every remaining difference
was an explicit anchor, which is strong evidence for the character class and for
the dedup convention. But that harness could not see three of the five rules at
all: the harvest normalises whitespace when it strips tags, so double spaces and
non-breaking spaces never reached it, and the trim question happened not to occur
in the sample. Those three came from looking up the anchor Hugo actually emitted
for one specific heading each. A sample that scores well can still be blind to a
whole rule.
Two cross-language traps sit in here. JavaScript's \s matches a non-breaking
space and Go's does not, so the obvious \s in the TypeScript version hyphenated a
character Hugo discards. And \p{N} includes Unicode "other numbers" such as
subscripts, where Hugo keeps only decimal digits, so the class has to be \p{Nd}.
Neither shows up until a page happens to use one.
The table of contents slugs the whitespace-trimmed raw heading rather than the
rendered title, because rendering collapses whitespace runs that Hugo's anchor
preserves. That is safe only because no heading in this repo contains a Markdown
link, which is the one case where raw and rendered would diverge; 31 headings
contain a double space, so the trade is clearly the right way round.
Learned: a sample can validate an algorithm and still be blind to whole rules -- the 3,616-pair harvest could not show the trim, whitespace-run or non-breaking-space behaviour because harvesting normalises whitespace, so each of those three came from looking up the anchor Hugo emitted for one heading
Learned: JavaScript's \s matches a non-breaking space where Go's does not, and \p{N} admits subscripts where Hugo allows only decimal digits, so a slug ported between the two languages diverges on characters that no test page happens to contain
Constraint: slugify in build/transform_json_sections.ts and the id block in layouts/partials/toc-from-markdown.html must stay equivalent -- keep underscores, keep hyphen runs, do not collapse whitespace runs, do not trim leading or trailing hyphens, and admit only letters, decimal digits, ASCII whitespace, underscore and hyphen
Constraint: slug the whitespace-trimmed raw heading, never the rendered title -- rendering collapses whitespace runs that Hugo's anchor keeps, and trimming whitespace is not the same as trimming hyphens
Rejected: trimming leading and trailing hyphens from the slug | Hugo does not, and it broke every heading ending in punctuation
Rejected: slugging the rendered title for consistency with the displayed text | it collapses the whitespace runs Hugo's anchor preserves, and 31 headings depend on that
Gaps: two table-of-contents entries still miss, each for its own non-slug reason -- develop/clients/observability has a "## Tracing overview" heading that reaches no anchor in the rendered HTML at all, and on operate/rc/databases/migrate-databases the dedup suffixes an explicit anchor because an earlier heading's derived slug already claimed the name, which is a precedence question rather than an algorithm one
Ticket: DOC-6939
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history: Memory updated at 3436e62 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3436e62. Configure here.
This was referenced Aug 6, 2026
andy-stark-redis
added
infrastructure
AI-friendliness
Contributor
Author
|
Thanks @dwdougherty ! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the metadata table of contents corrupting code-like headings — item D3 of DOC-6939, found while working the earlier items in #3754.
The defect
layouts/partials/toc-from-markdown.htmlstripped inline code markers first and then treated any remaining*or_as markdown emphasis. By that point nothing could tell that a character had come from inside a code span, so every asterisk and underscore in a code-formatted heading was eaten. 841 entries across 263 pages were affected, and the result is published API signatures that are simply wrong:`class AsyncSearchIndex(schema, *, redis_url=None, redis_client=None, connection_kwargs=None, validate_on_load=False, **kwargs)`class AsyncSearchIndex(schema, , redisurl=None, redisclient=None, connectionkwargs=None, validateon_load=False, *kwargs)class AsyncSearchIndex(schema, *, redis_url=None, redis_client=None, connection_kwargs=None, validate_on_load=False, **kwargs)The bare
*vanished,**kwargsbecame*kwargs, and underscores paired up and deleted. Mostly redisvl API reference pages, plus prose headings naming a snake_case identifier — "Control Re-Embedding Withskip_embedding_if_present" became "…With skipembeddingif_present".sections[]in the JSON feed was never affected; this is confined totableOfContentsin the per-page Markdown metadata block.The fix
Title flattening now goes through
RenderString+plainifyinstead of four hand-rolled regexes, so Goldmark's actual CommonMark rules apply — including the ban on intraword underscore emphasis, which is whyredis_urlnow survives whether or not it sits in a code span. No regex reproduces that rule, which is why this hands the job over rather than patching the patterns.Two Hugo behaviours worth knowing if you review this closely:
markdownifyrenders the string as a block. A heading beginning1.parses as an ordered list and loses its number. 299 headings start that way, and Hugo's anchor includes the number, so dropping it broke the deep link as well as the title.display: "inline"does not prevent that. Results were byte-identical tomarkdownify. The marker has to be escaped before rendering, which is what the code now does.Two smaller changes came with it:
{#redis.error_reply}, giving an id ofredis.errorreplyand a dead deep link on eightlua-apientries. It now lifts the anchor off the raw heading first, matching whattransform_json_sections.tsdoes.<a id="…"></a>tag rather than Hugo's brace syntax. The renderer correctly strips it, which left them as the only two ids that got worse. Converting them to{#…}was cheaper than teaching the partial a second anchor form for two headings, and the anchor value is unchanged so existing links still resolve.Verification
Scored by building before and after and comparing every TOC entry against the anchors in the same rendered HTML, since this change doesn't touch HTML output.
One caveat for anyone re-running it: the single remaining apparently-mangled entry is not a defect. A write-behind heading escapes its own backticks, so they are literal characters and the rendered title correctly keeps them — matching the rendered page heading.
What a reviewer should focus on
Not in this PR
Aligning
slugifywith Goldmark's anchor algorithm, which would close the remaining section-id gap (91.9%, untouched here) — that's the C7 remainder on the ticket and moves ids far more widely.Also newly noted, not fixed:
sections[].titlestill holds raw Markdown, so it no longer string-matches the rendered TOC title. Neither is wrong, but they've diverged in representation, and the transform can't call Goldmark.🤖 Generated with Claude Code
Note
Medium Risk
User-visible metadata TOC titles and deep-link ids change widely (~900+ titles); logic is concentrated in the Hugo partial and shared slugify, but behavior is validated against rendered HTML anchors.
Overview
Fixes metadata
tableOfContentstitles that were wrong on code-heavy headings (e.g. redisvl API signatures): title flattening now usesRenderString+plainifyinstead of regex that stripped backticks first and then mangled*/_inside signatures.toc-from-markdown.htmlalso parses{#explicit-id}from the raw heading before any cleaning (fixes anchors like{#redis.error_reply}), escapes leading1.list markers so numbered headings keep their prefix, and derives section ids from the raw heading with Goldmark’s github slug rules—kept in sync withslugifyinbuild/transform_json_sections.ts.Two RDI Aurora/RDS headings switch from inline
<a id="…">to{#…}so ids stay stable. Expect many published TOC titles (and some ids) to change; deep-link accuracy improves.Reviewed by Cursor Bugbot for commit 3436e62. Bugbot is set up for automated code reviews on this repo. Configure here.