From c2c9c0be5f8c6fc0868c9492e74d0ecae7653ebd Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 6 Aug 2026 11:19:14 +0100 Subject: [PATCH 1/2] DOC-6939 Render TOC titles with Hugo instead of hand-rolled regexes The metadata table of contents stripped inline code markers first and then treated any remaining asterisk or underscore as emphasis, so by that point nothing could tell that a character had come from inside a code span. 841 entries across 263 pages lost characters, publishing incorrect API signatures on the redisvl reference pages -- "**kwargs" arrived as "*kwargs" and "redis_url" as "redisurl". Title flattening now goes through RenderString and plainify, so Goldmark's actual CommonMark rules apply, including the ban on intraword underscore emphasis, which is why snake_case now survives whether or not it sits in a code span. No regex reproduces that rule, which is the reason for handing the job over rather than patching the patterns. Two things about the Hugo side are worth knowing before touching this. markdownify renders the string as a block, so a heading beginning "1. " parses as an ordered list and loses its number -- 299 headings start that way, and Hugo's anchor includes the number, so dropping it broke the deep link as well as the title. Switching to RenderString with display "inline" made no difference at all: the results were byte-identical, so "inline" does not stop block parsing. Escaping the marker before rendering does. The explicit-anchor extraction also turned out to be in the wrong place. It ran after the title cleaning, so the old regexes ate the underscores out of anchors such as the one on redis.error_reply, yielding an id of redis.errorreply and a dead deep link on eight lua-api entries. It now lifts the anchor off the raw heading before anything touches it, which is also what the TypeScript side does. Two headings on the RDI Aurora page used a raw HTML anchor tag rather than Hugo's brace syntax, which the renderer correctly strips, leaving them as the only two ids that got worse. Converting them to brace syntax was cheaper than teaching the partial a second anchor form for two headings, and the anchor value is unchanged so existing links still resolve. Mangled entries drop from 841 to zero. Titles change on 914 entries, 858 of them regaining characters. Of the 20 ids that change, all 20 now match the anchor Hugo renders and none regresses, taking TOC id parity from 94.46% to 94.51% measured over all 37,566 entries rather than a sample. Build time is unchanged at about two and a quarter minutes, so rendering 41,000 headings costs nothing measurable. One measurement caveat for anyone re-running this: the last apparently-mangled entry is not a defect. A write-behind heading escapes its own backticks, so they are literal characters and the rendered title keeps them, matching the HTML. Learned: RenderString with display "inline" does not stop Goldmark parsing block structures -- a heading beginning "1. " still becomes an ordered list and loses its number, byte-identically to markdownify, so the marker has to be escaped before rendering Constraint: keep the leading ordered-list marker escaped before rendering, or 299 headings lose their number and stop matching the anchor Hugo generates for them Constraint: lift the explicit {#anchor} off the raw heading before any rendering or cleaning touches it, or whatever runs first gets to mangle the anchor and the deep link dies silently Rejected: markdownify for title flattening | it renders the string as a block, so leading list markers and blockquote markers are consumed Rejected: skipping emphasis stripping when the heading contains a backtick | it leaves snake_case mangled outside code spans and reimplements CommonMark's intraword rule by hand Gaps: sections[].title still holds raw Markdown, so it no longer string-matches the rendered TOC title, and section-id anchor parity is untouched at 91.9% pending the slug alignment in C7 Ticket: DOC-6939 Co-Authored-By: Claude Opus 5 (1M context) --- .../aws-aurora-rds/aws-aur-mysql.md | 4 +- layouts/partials/toc-from-markdown.html | 48 +++++++++++++------ 2 files changed, 35 insertions(+), 17 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md index 1e9d7e1b05..8299b1948b 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md @@ -39,7 +39,7 @@ To add a reader node to an existing database, select **Add reader** from the **A You can also create one during database creation by selecting **Create an Aurora Replica or Reader node in a different AZ (recommended for scaled availability)** under **Availability & durability > Multi-AZ deployment**. -## Create and apply parameter group +## Create and apply parameter group {#aurora-create-and-apply-parameter-group} RDI requires some changes to database parameters. On AWS Aurora, you change these parameters via a parameter group. @@ -108,7 +108,7 @@ RDI requires some changes to database parameters. On AWS Aurora, you change thes - [ ] [Create Debezium user](#rds-create-debezium-user) ``` -## Create and apply parameter group +## Create and apply parameter group {#rds-create-and-apply-parameter-group} RDI requires some changes to database parameters. On AWS RDS, you change these parameters via a parameter group. diff --git a/layouts/partials/toc-from-markdown.html b/layouts/partials/toc-from-markdown.html index f59a6e6be5..fb63bbcdcd 100644 --- a/layouts/partials/toc-from-markdown.html +++ b/layouts/partials/toc-from-markdown.html @@ -2,6 +2,7 @@ {{- /* This parses ## and ### headers directly from .RawContent */ -}} {{- /* since Hugo's .TableOfContents may be empty for custom templates */ -}} +{{- $page := . -}} {{- $content := .RawContent -}} {{- /* Unescape HTML entities that may be present in .RawContent */ -}} @@ -47,28 +48,45 @@ {{- $title = replaceRE `^## +` "" $headerMatch -}} {{- end -}} - {{- /* Remove inline code markers */ -}} - {{- $title = replaceRE "`([^`]+)`" "$1" $title -}} - {{- /* Remove other inline formatting */ -}} - {{- $title = replaceRE `\*\*([^*]+)\*\*` "$1" $title -}} - {{- $title = replaceRE `\*([^*]+)\*` "$1" $title -}} - {{- $title = replaceRE `_([^_]+)_` "$1" $title -}} - {{- /* Trim whitespace */ -}} - {{- $title = $title | strings.TrimSpace -}} - - {{- /* Hugo's explicit heading anchor, "## Title {#custom-id}". Hugo uses the - anchor as the rendered heading id and does not display it, so it has to come - out of the title and be used verbatim as the id. Matched strictly, because a - loose trailing-brace match would eat Python signatures ending in an empty - dict and dict defaults such as {'extra': 'ignore'}. */ -}} + {{- /* Hugo's explicit heading anchor, "## Title {#custom-id}". Hugo uses the anchor as the + rendered heading id and does not display it, so it has to come out of the title and + be used verbatim as the id. Matched strictly, because a loose trailing-brace match + would eat Python signatures ending in an empty dict and dict defaults such as + {'extra': 'ignore'}. + + Lifted off the RAW heading, before any rendering. Doing it afterwards means the + anchor has to survive whatever the renderer does to it, and the emphasis regexes + this used to sit behind ate the underscores out of anchors such as + {#redis.error_reply}, yielding redis.errorreply and a dead deep link. */ -}} {{- $explicitId := "" -}} {{- with (findRESubmatch `\s*\{#([A-Za-z0-9][A-Za-z0-9_.:-]*)\}\s*$` $title 1) -}} {{- $explicitId = index (index . 0) 1 -}} {{- end -}} {{- if $explicitId -}} - {{- $title = $title | replaceRE `\s*\{#[A-Za-z0-9][A-Za-z0-9_.:-]*\}\s*$` "" | strings.TrimSpace -}} + {{- $title = $title | replaceRE `\s*\{#[A-Za-z0-9][A-Za-z0-9_.:-]*\}\s*$` "" -}} {{- end -}} + {{- /* Escape a leading ordered-list marker before rendering. Even with display "inline" + the renderer reads "1. Launch the app" as a list item and drops the number, where + the rendered page heading keeps it -- and Hugo's anchor includes it, so dropping it + breaks the deep link too. 299 headings start this way; none starts with -, + or *. */ -}} + {{- $title = $title | replaceRE `^(\d+)([.)])(\s)` `${1}\${2}${3}` -}} + + {{- /* Reduce the heading to plain text with Hugo's own Markdown renderer rather than + hand-rolled regexes. + + What this replaces stripped inline code markers first and then treated any remaining + * or _ as emphasis, so by that point there was no way to tell that a character had + come from inside a code span. Code-formatted headings therefore lost characters: + "**kwargs" became "*kwargs" and "redis_url" became "redisurl", which published + incorrect API signatures on the redisvl reference pages. + + Goldmark applies the real CommonMark rules, including the ban on intraword underscore + emphasis, so snake_case survives whether or not it sits in a code span. RenderString + rather than markdownify because markdownify renders the string as a block. plainify + then strips the inline markup and htmlUnescape restores literal text. */ -}} + {{- $title = $page.RenderString (dict "display" "inline") $title | plainify | htmlUnescape | strings.TrimSpace -}} + {{- /* Generate ID (slug) from title */ -}} {{- $id := $title | lower -}} {{- /* Replace spaces with hyphens */ -}} From 3436e6293cd36178baa4ea367d99ffcb4355a749 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 6 Aug 2026 11:45:12 +0100 Subject: [PATCH 2/2] DOC-6939 Align both slug implementations with Hugo's heading anchors Section ids now match the anchor Hugo renders on the page, so url#section-id deep links resolve. Parity goes from 79.6% before this branch, through 91.9% once explicit anchors were parsed, to 100.0% over the 2,594 sections in a 400-page sample. The metadata table of contents goes from 94.51% to 99.995% over all 37,566 entries. Hugo has no autoHeadingID setting, so it uses Goldmark's default github style. Five rules make it up and all five were wrong before. Underscores are kept, so redis_url stays redis_url instead of collapsing to redis-url. Runs of hyphens are kept. Whitespace runs are not collapsed, so "The special $ ID" anchors as the-special--id, the discarded dollar leaving the spaces either side of it. Leading and trailing hyphens are not trimmed: "Negation !" really does anchor as "negation-" and "- and + special IDs" as "--and--special-ids". And only ASCII whitespace and decimal digits count, so a non-breaking space is discarded rather than hyphenated, and the subscripts in "Naming convention: LVQx" vanish. Worth knowing how those rules were established, because it was not all from the same evidence. Harvesting 3,616 heading-and-anchor pairs out of the rendered HTML took the algorithm from 80.2% to 90.3% and showed that every remaining difference was an explicit anchor, which is strong evidence for the character class and for the dedup convention. But that harness could not see three of the five rules at all: the harvest normalises whitespace when it strips tags, so double spaces and non-breaking spaces never reached it, and the trim question happened not to occur in the sample. Those three came from looking up the anchor Hugo actually emitted for one specific heading each. A sample that scores well can still be blind to a whole rule. Two cross-language traps sit in here. JavaScript's \s matches a non-breaking space and Go's does not, so the obvious \s in the TypeScript version hyphenated a character Hugo discards. And \p{N} includes Unicode "other numbers" such as subscripts, where Hugo keeps only decimal digits, so the class has to be \p{Nd}. Neither shows up until a page happens to use one. The table of contents slugs the whitespace-trimmed raw heading rather than the rendered title, because rendering collapses whitespace runs that Hugo's anchor preserves. That is safe only because no heading in this repo contains a Markdown link, which is the one case where raw and rendered would diverge; 31 headings contain a double space, so the trade is clearly the right way round. Learned: a sample can validate an algorithm and still be blind to whole rules -- the 3,616-pair harvest could not show the trim, whitespace-run or non-breaking-space behaviour because harvesting normalises whitespace, so each of those three came from looking up the anchor Hugo emitted for one heading Learned: JavaScript's \s matches a non-breaking space where Go's does not, and \p{N} admits subscripts where Hugo allows only decimal digits, so a slug ported between the two languages diverges on characters that no test page happens to contain Constraint: slugify in build/transform_json_sections.ts and the id block in layouts/partials/toc-from-markdown.html must stay equivalent -- keep underscores, keep hyphen runs, do not collapse whitespace runs, do not trim leading or trailing hyphens, and admit only letters, decimal digits, ASCII whitespace, underscore and hyphen Constraint: slug the whitespace-trimmed raw heading, never the rendered title -- rendering collapses whitespace runs that Hugo's anchor keeps, and trimming whitespace is not the same as trimming hyphens Rejected: trimming leading and trailing hyphens from the slug | Hugo does not, and it broke every heading ending in punctuation Rejected: slugging the rendered title for consistency with the displayed text | it collapses the whitespace runs Hugo's anchor preserves, and 31 headings depend on that Gaps: two table-of-contents entries still miss, each for its own non-slug reason -- develop/clients/observability has a "## Tracing overview" heading that reaches no anchor in the rendered HTML at all, and on operate/rc/databases/migrate-databases the dedup suffixes an explicit anchor because an earlier heading's derived slug already claimed the name, which is a precedence question rather than an algorithm one Ticket: DOC-6939 Co-Authored-By: Claude Opus 5 (1M context) --- build/transform_json_sections.ts | 31 +++++++++++++++++++-- layouts/partials/toc-from-markdown.html | 37 ++++++++++++++++++------- 2 files changed, 56 insertions(+), 12 deletions(-) diff --git a/build/transform_json_sections.ts b/build/transform_json_sections.ts index 38721b71aa..daf7512007 100644 --- a/build/transform_json_sections.ts +++ b/build/transform_json_sections.ts @@ -111,11 +111,38 @@ function parseHeading(raw: string): { title: string; id: string } { return { title: raw, id: slugify(raw) }; } +/** + * Slugify a heading the way Hugo does, so section ids match the anchor on the page. + * + * Hugo has no autoHeadingID setting in config.toml, so it uses Goldmark's default + * "github" style: lowercase, discard anything that is not a letter, decimal digit, ASCII + * whitespace, underscore or hyphen, then turn each remaining whitespace character into one + * hyphen. Note decimal digit specifically -- a subscript such as the one in + * "Naming convention: LVQx" is a Unicode "other number" and Hugo discards it, + * anchoring as "naming-convention-lvqbxb". + * + * Five details matter and all five were wrong before. Underscores and runs of hyphens + * are KEPT, so "redis_url" stays "redis_url" rather than collapsing to "redis-url". + * Whitespace runs are NOT collapsed, so "The special $ ID" becomes "the-special--id" + * with two hyphens, the discarded "$" leaving the spaces either side of it. Leading and + * trailing hyphens are NOT trimmed -- a heading "Negation !" really does anchor as + * "negation-" and "- and + special IDs" as "--and--special-ids". And the whitespace + * class is ASCII only, matching Go's \s, so a non-breaking space is discarded rather + * than turned into a hyphen: "Development environment" anchors as + * "developmentenvironment". JavaScript's \s would have hyphenated it. + * + * Measured against 3,616 heading/anchor pairs harvested from the rendered HTML: this + * reproduces Hugo's anchor for 90.3% of them, against 80.2% for the previous version, + * and every remaining difference is a heading carrying an explicit {#anchor}, which + * parseHeading handles before this function is reached. The trim, whitespace-run and + * non-breaking-space rules were each confirmed against the anchor Hugo emitted for a + * specific heading, since the harvested sample normalises whitespace and cannot show them. + */ function slugify(text: string): string { return text .toLowerCase() - .replace(/[^a-z0-9]+/g, '-') - .replace(/^-|-$/g, ''); + .replace(/[^\p{L}\p{Nd}\t\n\f\r _-]/gu, '') + .replace(/[\t\n\f\r ]/g, '-'); } // Section IDs to filter out (metadata noise, not useful for RAG) diff --git a/layouts/partials/toc-from-markdown.html b/layouts/partials/toc-from-markdown.html index fb63bbcdcd..07c0af5aeb 100644 --- a/layouts/partials/toc-from-markdown.html +++ b/layouts/partials/toc-from-markdown.html @@ -85,18 +85,35 @@ emphasis, so snake_case survives whether or not it sits in a code span. RenderString rather than markdownify because markdownify renders the string as a block. plainify then strips the inline markup and htmlUnescape restores literal text. */ -}} + {{- /* Keep the raw heading for the id, because Hugo slugs the heading's own text, which + preserves whitespace runs where rendering collapses them: a heading with a double + space anchors as "choosing-the-capacity--capacity", and slugging the rendered title + would give one hyphen. Slugging the raw text costs nothing, because the markup + characters it still contains are discarded by the slug rules anyway, and no heading + in this repo contains a Markdown link -- the one case where the two would diverge. + + Whitespace-trimmed, but hyphens are not: a trailing space would otherwise become a + trailing hyphen, where Hugo slugs the trimmed heading text. Those are not the same + thing -- "Negation !" trims to itself, and the space left behind when the "!" is + discarded still becomes the trailing hyphen Hugo emits. */ -}} + {{- $rawTitle := $title | strings.TrimSpace -}} + {{- $title = $page.RenderString (dict "display" "inline") $title | plainify | htmlUnescape | strings.TrimSpace -}} - {{- /* Generate ID (slug) from title */ -}} - {{- $id := $title | lower -}} - {{- /* Replace spaces with hyphens */ -}} - {{- $id = $id | replaceRE `\s+` "-" -}} - {{- /* Remove special characters except hyphens */ -}} - {{- $id = $id | replaceRE `[^a-z0-9-]` "" -}} - {{- /* Remove consecutive hyphens */ -}} - {{- $id = $id | replaceRE `-+` "-" -}} - {{- /* Trim leading/trailing hyphens */ -}} - {{- $id = $id | replaceRE `^-+|-+$` "" -}} + {{- /* Slug the title the way Hugo does, so the id matches the anchor on the page. + Goldmark's default "github" style: lowercase, discard anything that is not a + letter, number, whitespace, underscore or hyphen, then turn each remaining + whitespace character into one hyphen. + + Underscores and runs of hyphens are kept, whitespace runs are NOT collapsed, and + leading and trailing hyphens are NOT trimmed -- "Negation !" really does anchor as + "negation-". The previous version stripped underscores, collapsed runs and trimmed, + so it disagreed with the page anchor three ways. Slugged from the raw heading, not + the rendered title, and kept identical to slugify in + build/transform_json_sections.ts -- change the two together. */ -}} + {{- $id := $rawTitle | lower -}} + {{- $id = $id | replaceRE `[^\p{L}\p{Nd}\s_-]` "" -}} + {{- $id = $id | replaceRE `\s` "-" -}} {{- /* An explicit anchor overrides the slug entirely */ -}} {{- if $explicitId -}}