✨ feat(serialize): lossless byte-preserving to_source#600
Merged
Conversation
Node.to_source() re-emits a parsed tree from its original source bytes, copying the verbatim span of every element and text run the parse left untouched and reserializing only the nodes a mutation changed. On a tree parsed with source_locations=True, an unedited round trip reproduces the input byte for byte -- author quoting, tag-name case, character-reference spelling, and insignificant whitespace intact -- for markup that parsed without implied elements or content reordering. After an edit only the changed node's markup is rewritten while every untouched sibling and subtree keeps its original span. The walk composes per node from the spans a source-location parse records: an element copies its start-tag span, recurses, and copies its end-tag span; a text run copies its zero-copy slice while it is still one. Inserted and removed nodes are reflected by reading the current tree; a changed attribute value, which a span cannot reveal, is flagged on the element's start tag by the mutation API so that one tag rebuilds from the current attributes. The walk is iterative, so an arbitrarily deep tree never grows the C stack. This is the tree-based counterpart to the streaming turbohtml.rewrite, the model Cloudflare's lol-html popularized. Adds a lossless-serialize benchmark op wired through the pyperf, CodSpeed, and PGO corpora, the four Diataxis docs, and an extension of the lol-html migration guide. closes tox-dev#547
Merging this PR will not alter performance
Performance Changes
Comparing 
Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The parse-then-serialize round trip normalizes the whole document: it re-quotes every attribute, lowercases tag names, and rewrites character references to their canonical form. That is the right default for canonical, diff-stable output, but it is the wrong tool for a surgical edit, where you want to change one thing and leave every other byte, including the author's own quoting and formatting, exactly as written. This closes #547.
Node.to_source()fills that gap. It re-emits a parsed tree from its original source bytes, copying the verbatim span of every element and text run the parse left untouched and reserializing only the nodes a mutation changed. Parse withsource_locations=Trueand an unedited round trip reproduces the input byte for byte for markup that parsed without implied elements or content reordering; after an edit only the changed node's markup is rewritten while every untouched sibling and subtree keeps its original span. 🧩 It is the tree-based counterpart to the streamingturbohtml.rewrite, the model Cloudflare's lol-html popularized: reach for the rewriter at the edge and for a page larger than memory, and forto_sourcewhen the edit needs a query the stream cannot decide.The C serializer walks the tree once, composing each node from the per-element spans a source-location parse already records. An element copies its start-tag span, recurses into its children, and copies its end-tag span; a text run copies its zero-copy slice while it is still one. Because contiguous child spans tile the parent's content, concatenating the pieces reproduces the original without a separate whole-subtree copy, and each clean node costs one
memcpy. Inserted and removed nodes are reflected by reading the tree's current state; the one edit a span cannot reveal, a changed attribute value, is flagged on the element's start tag by the mutation API so that one tag rebuilds from the current attributes while its end tag, children, and siblings stay verbatim. The walk is iterative, so an arbitrarily deep tree never grows the C stack.Two normalizations the parser itself performs are not reversible from the tree and so are not preserved: a lowercase
<!doctype html>re-emits as<!DOCTYPE html>, and a character reference the tokenizer resolved re-emits in its canonical spelling (&,<,>, and survive because the escaper re-encodes them; a legacy or numeric reference becomes its character). Where every byte of an error-recovering or reordering parse must survive, the streaming rewriter, which never discards the token stream, remains the tool; the serialization explanation states the full round-trip contract.Alongside the C implementation and its thin
to_sourcemethod, this adds alossless-serializebenchmark op wired through the pyperf, CodSpeed, and PGO corpora, the four Diataxis docs, and a new section of the lol-html migration guide covering when a tree beats the stream. The migration guide keeps its existing rationale for carrying nobench-tableorpackage-meta: lol-html is a Rust and WebAssembly project with no in-process Python peer to benchmark and no PyPI package the badge directive can describe.