Iterate README renderer pipeline#462
Merged
Merged
Conversation
Avoid the jsdom DOMParser and DOMPurify string sanitization paths in render_readmes.mjs. Those paths retained substantial DOM state during a cold README render and grew heap roughly linearly with package count. Profiling the old implementation on 4,892 READMEs showed a peak working set around 3.5 GB and a runtime around 30 seconds. The patched version uses detached template fragments for post-processing and asks DOMPurify for a DOM fragment before serializing it back to HTML. The same cold render now peaks around 550 MB and runs in about 18 seconds. That is roughly 84% less peak memory and 39% faster for the current corpus. The remaining growth is small enough that 10k packages should stay well below the default Node heap limit, instead of risking an out-of-memory failure around 6k-7k. Security behavior is preserved: DOMPurify is still the final sanitizer, with the same default policy. Regression checks rendered all 4,892 README files byte-for-byte identically to the existing readmes_rendered.json, and static/readme-renderer.test.mjs passed.
The readme rendering is slow and memory intensive, but there's no reason to redo all that work every time. We now store a hash of the text used to render the readme inside readmes_renderes.json to avoid re-rendering.
Keep the per-README cache hash focused on the source README text, and record the renderer context once in readmes_rendered.json as the __environment entry. The renderer environment hash includes render_readmes.mjs, the shared README renderer, this cache helper, and package-lock.json. If any of those inputs change, the prerender file is treated as coming from a changed rendering environment and all entries are re-computed.
Restore readmes_rendered.json before running the build so render_readmes.mjs can reuse unchanged entries across workflow runs. Use a broad rolling Actions cache key and leave cache correctness to the script's per-entry hashes.
Check the rendered README environment marker when Eleventy starts. If the prerendered file was built with a different renderer environment, discard it instead of injecting stale cached HTML into package pages. This keeps renderer hacking honest: local builds and the dev server fall back to the existing browser fetch/render path until readmes_rendered.json is regenerated with the current renderer.
kaste
enabled auto-merge
July 22, 2026 11:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Obviously we had a bug here, and memory growth was 600kB per rendered README.
It is important to tackle this besides the #461 because we always have fresh/stale
states at some point. And with that growth, we quickly can't produce the initial
render without going out-of-memory.
But that over cached clearly as it only invalidates the cache if the source, the
raw markdown text, changed. Obviously, our feature set, the renderer itself,
can change and must invalidate the previously rendered files.
The third commit tries to tackle that.
Finally, use the feature set in the CI pipeline.
Convenient: also check the cached READMEs during dev. No surprises if we ever hack on the renderer again.
Closes #461