Skip to content

DOC-6873 Fix slow Hugo builds#3686

Merged
andy-stark-redis merged 1 commit into
mainfrom
DOC-6873-fix-slow-hugo-build
Jul 22, 2026
Merged

DOC-6873 Fix slow Hugo builds#3686
andy-stark-redis merged 1 commit into
mainfrom
DOC-6873-fix-slow-hugo-build

Conversation

@andy-stark-redis

@andy-stark-redis andy-stark-redis commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

What & why

The Hugo build had regressed to ~10+ minutes. Investigation (hugo --templateMetrics) traced it to layouts/partials/tabs/source.html re-running Chroma over the entire source file on every slice, every step, and every page — 3,000+ uncached highlights, the single most expensive thing in the build.

Two changes:

  1. Cache the full-file highlight. New layouts/partials/tabs/highlight-file.html does the read+highlight and is called via partialCached keyed on (file, language, options), so each file is highlighted once per build instead of once per slice/step/page. source.html now slices the cached HTML.

    • 95% cache-hit rate, syntax-highlight user-CPU down ~27% (4m31s → 3m19s).
    • Proven byte-neutral on every content page (fixed-vs-baseline diffs equal the build's own run-to-run noise floor).
  2. Fix a pre-existing nondeterminism it uncovered. The tce-fullsrc full-file <template> dedup in tabbed-clients-example.html tracks its "seen" set in .Page.Store, which persists across a page's output-format renders. Hugo's internal RSS renders each item's full .Content, re-executing the page's shortcodes in a second render that shares that store and races the HTML render. On the heavy develop/data-types/streams/ page this meant the served HTML intermittently shipped 8 or 0 full-source templates (i.e. the "view full file" affordance was randomly broken). New layouts/_default/rss.xml sources item descriptions from front matter instead of rendering .Content, so content shortcodes never run during RSS. Emission is now deterministic for every page.

⚠️ Reviewer callout — RSS behaviour change

RSS feeds now carry each item's front-matter description (falling back to title) instead of the full rendered page HTML. This is the mechanism of fix #2 (and is normal/lighter for feeds), but it is an outward-facing change to feed content — flagging it explicitly.

Verification

Two consecutive full builds, exit 0, no template errors:

build 1 build 2
streams full-source templates 8 8 (deterministic)
sorted-sets (control) 11 11 (unchanged)
data-types RSS items / rendered-code-in-RSS 7 / 0 7 / 0
RSS well-formed (xmllint) OK

Wall-clock speedup is not claimed as a hard number: the benchmark machine was I/O-bound (builds swung 5–16 min dominated by system time), so only the user-CPU reduction is trustworthy.

Ticket: DOC-6873

🤖 Generated with Claude Code


Note

Medium Risk
Build output and RSS item descriptions change in user-visible ways (feeds lose full HTML bodies); HTML for examples is intended to stay byte-neutral but the RSS behaviour is an explicit outward-facing change.

Overview
Speeds up Hugo builds by caching full-file Chroma output for tabbed client examples instead of re-highlighting the same source on every slice, step, and page.

tabs/source.html now pulls a single highlighted file via partialCached on new tabs/highlight-file.html (keyed by path, language, and options) and only slices that HTML when a line range is needed. Fixes intermittent missing “view full file” templates on pages with many clients-example shortcodes.

A custom layouts/_default/rss.xml stops rendering each item’s full .Content in the RSS pass. Item descriptions come from front matter (description or title) only, so RSS no longer re-runs those shortcodes and races HTML for .Page.Store (tceFullKeys). RSS feed text changes from full page HTML to short summaries—that’s intentional and the mechanism of the determinism fix.

Reviewed by Cursor Bugbot for commit b249f9f. Bugbot is set up for automated code reviews on this repo. Configure here.

…ce dedup

A full build had regressed to 10+ minutes. tabs/source.html was re-running
Chroma over the entire source file on every slice, every step, and every page
(3000+ uncached highlights); moving that behind partialCached keyed on
file+language+options cuts syntax-highlight CPU by ~27% and is byte-neutral on
every stable page.

Chasing an apparent output change from that cache led to the real prize: the
full-source template embeds on the streams page were already nondeterministic on
main, shipping 8 or 0 at random. The cache was not the cause. The dedup that
emits each full file once per page tracks its "seen" set in .Page.Store, which
persists across a page's output-format renders; Hugo's internal RSS renders each
item's full .Content, re-executing the page's shortcodes in a second render that
shares that store and races the HTML render. Whichever ran first claimed the
emissions, so the served HTML sometimes got none. The caching change only
perturbed timing, which is why it first looked causal. The fix is a custom RSS
template that builds item descriptions from front matter instead of rendering
.Content, so content shortcodes never run during RSS. Emission is now
deterministic for every page, not just streams.

Learned: why the build was slow (uncached full-file Chroma) and why streams' full-source embeds were flaky (RSS shares .Page.Store with the HTML render)
Constraint: the RSS template must not render page .Content/.Summary; doing so re-executes content shortcodes in a second output-format render that shares .Page.Store and races the HTML render, intermittently dropping full-source templates
Rejected: per-output-format .Page.Store scoping | Hugo does not expose the current output format to shortcodes during content rendering
Rejected: ordinal-reset or stateless first-step dedup | unsafe under concurrent output-format renders, and a shortcode cannot know the page's other steps
Directive: keep the full-file Chroma highlight behind partialCached in tabs/source.html; do not inline highlight again (it re-highlights the whole file per slice, step, and page)
Gaps: wall-clock speedup unverified (builds were I/O-bound on a loaded machine, only the ~27% user-CPU drop is trustworthy); RSS confirmed as the second render but other output formats not exhaustively audited
Ticket: DOC-6873
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

DOC-6873

@andy-stark-redis andy-stark-redis changed the title DOC-6873 Cache tabbed-example highlighting; stop RSS racing full-source dedup DOC-6873 Fix slow Hugo builds Jul 21, 2026
@andy-stark-redis
andy-stark-redis requested a review from a team July 21, 2026 15:07
@andy-stark-redis andy-stark-redis self-assigned this Jul 21, 2026

@dwdougherty dwdougherty left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to test this locally, but I kept hitting rate limits on the client library sites. Approving anyway, because it's a new world now.

@andy-stark-redis

andy-stark-redis commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@dwdougherty You could do make localserve maybe, if you've got the repo folders set up? Thanks for the review, anyway :-)

@andy-stark-redis
andy-stark-redis merged commit d15bd90 into main Jul 22, 2026
78 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants