Skip to content

✨ feat(parser): add DOM-less SAX event parsing#575

Merged
gaborbernat merged 4 commits into
tox-dev:mainfrom
gaborbernat:feat/532-sax
Jul 7, 2026
Merged

✨ feat(parser): add DOM-less SAX event parsing#575
gaborbernat merged 4 commits into
tox-dev:mainfrom
gaborbernat:feat/532-sax

Conversation

@gaborbernat

Copy link
Copy Markdown
Member

Pulling a few facts out of a page and moving on had no first-class API. turbohtml.parse hands back a whole document you have to hold, and turbohtml.tokenize hands back raw tokens that know nothing about tree structure, so recovering the events a browser's parser fires (with implied html/head/body and foster-parented table content) meant walking a tree you did not want to keep. This adds turbohtml.saxparse, the event-driven, DOM-less parse issue #532 asked for. Subclass SaxHandler and pass it to sax_parse, or iterate iter_events for a stream of typed StartElement/EndElement/Characters/Comment/Doctype/ProcessingInstruction records.

The events come from the existing WHATWG tree builder, not a second parser. A new document-order walk in src/turbohtml/_c/tokenizer/sax.c drives th_tree_parse and streams the constructed tree with a single cursor and a phase bit, so the events carry the implied tags, foster parenting, and adoption-agency re-nesting that html.parser cannot reconstruct. 🌳 The walk builds no node object and returns no tree, so a one-pass extraction never pays for a document-sized Python object graph and keeps only what your handler keeps. Tokenization, construction, and the walk stay in C; the per-event dispatch is the one Python boundary.

WHATWG HTML has no processing instructions, so <?xml ...> parses as a bogus comment. The tokenizer now tags that token and the walk reports it as a ProcessingInstruction to match html.parser's handle_pi, while the DOM keeps storing a comment.

The explanation/sax note is honest about the limit: this will not parse a document larger than memory. A spec-correct tree builder cannot run in space proportional to the open-element depth, because the adoption agency and text coalescing move nodes the walk has already emitted, so the working tree survives until the parse ends. Against turbohtml.parse the gain is the object graph you never build and the tree you never keep. Against html.parser the gain is speed: a counting handler over sax_parse runs a few times faster than the same handler on html.parser, whose tokenizer and entity handling are pure Python, and it gets the corrected tree the standard library never builds. A sax benchmark op tracks this against html.parser on real pages, and the stdlib migration guide gains a section mapping each handle_* override onto its SaxHandler method.

@gaborbernat gaborbernat added documentation Improvements or additions to documentation enhancement New feature or request labels Jul 7, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 7, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 77 untouched benchmarks
🆕 1 new benchmark
⏩ 18 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
🆕 test_feature[sax] N/A 47.1 ms N/A

Comparing gaborbernat:feat/532-sax (3f2d1c4) with main (32c1c44)

Open in CodSpeed

Footnotes

  1. 18 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

gaborbernat and others added 4 commits July 7, 2026 04:02
Add turbohtml.saxparse: sax_parse fires SaxHandler callbacks and
iter_events yields typed records for each construct the parser builds,
without ever handing back a tree. The events come from the existing
WHATWG tree builder via a new document-order event walk in C, so they
reflect the constructed tree -- implied html/head/body, foster
parenting, the adoption agency -- not the raw token stream, which is
the one thing html.parser cannot give you. No per-node Python object is
created and nothing is retained after the parse.

A `<?...>` construct is a WHATWG bogus comment; the tokenizer now tags
that token so the walk can report it as a ProcessingInstruction,
matching html.parser's handle_pi, while the DOM still stores a comment.

closes tox-dev#532
clang-format wrapped sax_clear's Py_CLEAR onto two physical lines, which
orphaned its GCOVR_EXCL_BR_LINE marker from the macro's branch, so the
never-taken NULL arm was counted and the full-suite C branch-coverage
gate dropped to 99.9%. Bind the cast to a local so the macro and its
marker stay on one line, matching sax_traverse.
@gaborbernat gaborbernat merged commit 0fe506c into tox-dev:main Jul 7, 2026
48 checks passed
@gaborbernat gaborbernat deleted the feat/532-sax branch July 10, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant