✨ feat(rewrite): DOM-less streaming HTML rewriter (lol-html style)#591
Merged
gaborbernat merged 6 commits intoJul 7, 2026
Merged
Conversation
ddb9f94 to
c2fbcc8
Compare
Merging this PR will not alter performance
Performance Changes
Comparing Footnotes
|
a9658cd to
cec86fe
Compare
Add turbohtml.rewrite.rewrite(), a single-pass rewriter that transforms markup without building a tree. It streams the WHATWG tokenizer, tracks only the open-element stack, matches CSS selectors against that spine, and hands each match (and text/comment/doctype) to a Python handler that edits it in place, emitting the result incrementally. Working memory is O(open-element depth), and an untouched construct is reproduced verbatim. The engine reuses the existing tokenizer and native CSS selector engine. Because the pass never looks ahead, the streamable selector subset is the one decidable from an element and its ancestors; a sibling combinator, a positional or structural pseudo-class, or :has() is rejected at compile time. closes tox-dev#544
Add the rewrite op to the CodSpeed suite over a real page and classify it as an HTML-document op in the PGO training corpus.
rw_compile_rules takes a reference on each element handler with Py_NewRef, but rw_ctx_clear freed the compiled selectors without releasing those handlers, leaking one reference per rule on every rewrite() call. A leaked handler that closes over a stashed _RewriteHandle keeps that untracked wrapper alive to interpreter finalization. The live wrapper pins its heap type and, through it, the _html module, so CPython skips the module m_clear at shutdown. th_node_freelist_clear runs only from m_clear, so its pool-drain loop went uncovered and failed the C coverage gate on any job whose run left a handler leaked (node.c 41-43).
Newer clang and gcc instrument short-circuit, ternary, and macro branches that the 3.12 toolchain folds away, so the rewrite pass floored at 99.9% branch on 3.11-3.14 (both platforms) once the node.c line gap was fixed. Close every gap at the source: - fold rw_content_model's per-literal `len == N && memcmp` chain into a table + loop so one comparison branch replaces the unhittable operand fan-out - drop dead guards the tests can never take: the empty-name check in rw_tag_atom, the malloc `?: 1` on a tag name that is always >= 1, and the `name != NULL` arm of a custom attribute selector (always set) - replace a Py_CLEAR whose operand is provably non-NULL with an unconditional decref, and isolate the output-OOM check so only the unforceable branch carries the exclusion - add tests for the reachable paths: replace-then-remove, append-then set_content, empty attribute values, known-attribute removal, a non-ASCII tag, a void element inside dropped content, an overlong attribute name, and a second rule left unrun after the first raises rewrite.c reaches 100% line and branch under both llvm-cov and gcov.
cec86fe to
81c561d
Compare
th_node_freelist_clear runs only from the module m_clear slot at interpreter finalization, and only drains when the recycling pool is non-empty at that instant. Whether CPython reaches it with a populated pool is nondeterministic across interpreters: some 3.10/3.11 macOS and Linux runs finalize with the pool already emptied, leaving the drain body uncovered and the gate red at 99.9% at random. No portable test forces the state, so mark the loop GCOVR_EXCL and drop the sessionfinish priming that tried, unreliably, to seed it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Editing a few nodes in a large document meant parsing the whole thing into a tree first, which costs memory proportional to the document and is wasteful when the task is a forward transform: retag links, lazy-load images, strip trackers, redact text. This adds
turbohtml.rewrite.rewrite, a single-pass rewriter that transforms markup without building a tree, the model Cloudflare's lol-html uses to rewrite responses at the edge. Closes #544.The rewriter streams the existing WHATWG tokenizer over the input and keeps only the stack of currently-open elements. Each open element is a lightweight node whose sole live link is its parent, so the native CSS selector engine that powers
select()matches against that spine directly, and a matched element (or a text run, comment, or the doctype) is handed to a handler that edits it in place through anElementhandle: set or remove an attribute, insert markup around or inside it, replace its inner content, unwrap it, or drop it. 🚀 Working memory stays proportional to the open-element depth rather than the document size, so a page far larger than memory rewrites in a fixed footprint, and an untouched construct is copied through verbatim, so a rewrite that edits nothing returns its input unchanged.The one constraint the design imposes is the selector subset. A no-lookahead stream has seen an element's ancestors but not its following siblings or descendants, so only selectors decidable from an element and its ancestors match: type, universal, id, class, and attribute selectors, the descendant and child combinators,
:root, and:is()/:where()/:not()over that subset. A sibling combinator, a positional or structural pseudo-class (:nth-child,:only-child,:empty), or:has()is rejected at compile time withSelectorSyntaxError, matching lol-html's own restriction. Theexplanation/streamingnote covers why, and themigration/lol-htmlguide maps theHTMLRewriterandElementAPI across.The whole engine lives in the C extension (
_c/tokenizer/rewrite.c) reusing the tokenizer and selector engine; the Python layer is a thin typed shim. A depth guard bounds the open-element stack so a pathologically deep or unclosed input cannot exhaust memory or the C stack the matcher walks.