Skip to content

✨ feat(rewrite): DOM-less streaming HTML rewriter (lol-html style)#591

Merged
gaborbernat merged 6 commits into
tox-dev:mainfrom
gaborbernat:feat/544-streaming-rewriter
Jul 7, 2026
Merged

✨ feat(rewrite): DOM-less streaming HTML rewriter (lol-html style)#591
gaborbernat merged 6 commits into
tox-dev:mainfrom
gaborbernat:feat/544-streaming-rewriter

Conversation

@gaborbernat

Copy link
Copy Markdown
Member

Editing a few nodes in a large document meant parsing the whole thing into a tree first, which costs memory proportional to the document and is wasteful when the task is a forward transform: retag links, lazy-load images, strip trackers, redact text. This adds turbohtml.rewrite.rewrite, a single-pass rewriter that transforms markup without building a tree, the model Cloudflare's lol-html uses to rewrite responses at the edge. Closes #544.

The rewriter streams the existing WHATWG tokenizer over the input and keeps only the stack of currently-open elements. Each open element is a lightweight node whose sole live link is its parent, so the native CSS selector engine that powers select() matches against that spine directly, and a matched element (or a text run, comment, or the doctype) is handed to a handler that edits it in place through an Element handle: set or remove an attribute, insert markup around or inside it, replace its inner content, unwrap it, or drop it. 🚀 Working memory stays proportional to the open-element depth rather than the document size, so a page far larger than memory rewrites in a fixed footprint, and an untouched construct is copied through verbatim, so a rewrite that edits nothing returns its input unchanged.

The one constraint the design imposes is the selector subset. A no-lookahead stream has seen an element's ancestors but not its following siblings or descendants, so only selectors decidable from an element and its ancestors match: type, universal, id, class, and attribute selectors, the descendant and child combinators, :root, and :is()/:where()/:not() over that subset. A sibling combinator, a positional or structural pseudo-class (:nth-child, :only-child, :empty), or :has() is rejected at compile time with SelectorSyntaxError, matching lol-html's own restriction. The explanation/streaming note covers why, and the migration/lol-html guide maps the HTMLRewriter and Element API across.

The whole engine lives in the C extension (_c/tokenizer/rewrite.c) reusing the tokenizer and selector engine; the Python layer is a thin typed shim. A depth guard bounds the open-element stack so a pathologically deep or unclosed input cannot exhaust memory or the C stack the matcher walks.

@gaborbernat gaborbernat added documentation Improvements or additions to documentation enhancement New feature or request labels Jul 7, 2026
@gaborbernat gaborbernat force-pushed the feat/544-streaming-rewriter branch from ddb9f94 to c2fbcc8 Compare July 7, 2026 14:07
@codspeed-hq

codspeed-hq Bot commented Jul 7, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 82 untouched benchmarks
🆕 1 new benchmark
⏩ 18 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
🆕 test_feature[rewrite] N/A 7.8 ms N/A

Comparing gaborbernat:feat/544-streaming-rewriter (61379f9) with main (60cca4c)

Open in CodSpeed

Footnotes

  1. 18 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@gaborbernat gaborbernat force-pushed the feat/544-streaming-rewriter branch 4 times, most recently from a9658cd to cec86fe Compare July 7, 2026 20:33
Add turbohtml.rewrite.rewrite(), a single-pass rewriter that transforms
markup without building a tree. It streams the WHATWG tokenizer, tracks
only the open-element stack, matches CSS selectors against that spine,
and hands each match (and text/comment/doctype) to a Python handler that
edits it in place, emitting the result incrementally. Working memory is
O(open-element depth), and an untouched construct is reproduced verbatim.

The engine reuses the existing tokenizer and native CSS selector engine.
Because the pass never looks ahead, the streamable selector subset is the
one decidable from an element and its ancestors; a sibling combinator, a
positional or structural pseudo-class, or :has() is rejected at compile
time.

closes tox-dev#544
Add the rewrite op to the CodSpeed suite over a real page and classify it
as an HTML-document op in the PGO training corpus.
rw_compile_rules takes a reference on each element handler with Py_NewRef,
but rw_ctx_clear freed the compiled selectors without releasing those
handlers, leaking one reference per rule on every rewrite() call.

A leaked handler that closes over a stashed _RewriteHandle keeps that
untracked wrapper alive to interpreter finalization. The live wrapper
pins its heap type and, through it, the _html module, so CPython skips
the module m_clear at shutdown. th_node_freelist_clear runs only from
m_clear, so its pool-drain loop went uncovered and failed the C coverage
gate on any job whose run left a handler leaked (node.c 41-43).
Newer clang and gcc instrument short-circuit, ternary, and macro
branches that the 3.12 toolchain folds away, so the rewrite pass floored
at 99.9% branch on 3.11-3.14 (both platforms) once the node.c line gap
was fixed. Close every gap at the source:

- fold rw_content_model's per-literal `len == N && memcmp` chain into a
  table + loop so one comparison branch replaces the unhittable operand
  fan-out
- drop dead guards the tests can never take: the empty-name check in
  rw_tag_atom, the malloc `?: 1` on a tag name that is always >= 1, and
  the `name != NULL` arm of a custom attribute selector (always set)
- replace a Py_CLEAR whose operand is provably non-NULL with an
  unconditional decref, and isolate the output-OOM check so only the
  unforceable branch carries the exclusion
- add tests for the reachable paths: replace-then-remove, append-then
  set_content, empty attribute values, known-attribute removal, a
  non-ASCII tag, a void element inside dropped content, an overlong
  attribute name, and a second rule left unrun after the first raises

rewrite.c reaches 100% line and branch under both llvm-cov and gcov.
@gaborbernat gaborbernat force-pushed the feat/544-streaming-rewriter branch from cec86fe to 81c561d Compare July 7, 2026 20:59
th_node_freelist_clear runs only from the module m_clear slot at interpreter
finalization, and only drains when the recycling pool is non-empty at that
instant. Whether CPython reaches it with a populated pool is nondeterministic
across interpreters: some 3.10/3.11 macOS and Linux runs finalize with the pool
already emptied, leaving the drain body uncovered and the gate red at 99.9% at
random. No portable test forces the state, so mark the loop GCOVR_EXCL and drop
the sessionfinish priming that tried, unreliably, to seed it.
@gaborbernat gaborbernat merged commit 95dfb19 into tox-dev:main Jul 7, 2026
48 checks passed
@gaborbernat gaborbernat deleted the feat/544-streaming-rewriter branch July 10, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parser: DOM-less streaming rewriter

1 participant