✨ feat(parser): add DOM-less SAX event parsing#575
Merged
Conversation
Merging this PR will not alter performance
Performance Changes
Comparing Footnotes
|
Add turbohtml.saxparse: sax_parse fires SaxHandler callbacks and iter_events yields typed records for each construct the parser builds, without ever handing back a tree. The events come from the existing WHATWG tree builder via a new document-order event walk in C, so they reflect the constructed tree -- implied html/head/body, foster parenting, the adoption agency -- not the raw token stream, which is the one thing html.parser cannot give you. No per-node Python object is created and nothing is retained after the parse. A `<?...>` construct is a WHATWG bogus comment; the tokenizer now tags that token so the walk can report it as a ProcessingInstruction, matching html.parser's handle_pi, while the DOM still stores a comment. closes tox-dev#532
for more information, see https://pre-commit.ci
clang-format wrapped sax_clear's Py_CLEAR onto two physical lines, which orphaned its GCOVR_EXCL_BR_LINE marker from the macro's branch, so the never-taken NULL arm was counted and the full-suite C branch-coverage gate dropped to 99.9%. Bind the cast to a local so the macro and its marker stay on one line, matching sax_traverse.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pulling a few facts out of a page and moving on had no first-class API.
turbohtml.parsehands back a whole document you have to hold, andturbohtml.tokenizehands back raw tokens that know nothing about tree structure, so recovering the events a browser's parser fires (with impliedhtml/head/bodyand foster-parented table content) meant walking a tree you did not want to keep. This addsturbohtml.saxparse, the event-driven, DOM-less parse issue #532 asked for. SubclassSaxHandlerand pass it tosax_parse, or iterateiter_eventsfor a stream of typedStartElement/EndElement/Characters/Comment/Doctype/ProcessingInstructionrecords.The events come from the existing WHATWG tree builder, not a second parser. A new document-order walk in
src/turbohtml/_c/tokenizer/sax.cdrivesth_tree_parseand streams the constructed tree with a single cursor and a phase bit, so the events carry the implied tags, foster parenting, and adoption-agency re-nesting thathtml.parsercannot reconstruct. 🌳 The walk builds no node object and returns no tree, so a one-pass extraction never pays for a document-sized Python object graph and keeps only what your handler keeps. Tokenization, construction, and the walk stay in C; the per-event dispatch is the one Python boundary.WHATWG HTML has no processing instructions, so
<?xml ...>parses as a bogus comment. The tokenizer now tags that token and the walk reports it as aProcessingInstructionto matchhtml.parser'shandle_pi, while the DOM keeps storing a comment.The
explanation/saxnote is honest about the limit: this will not parse a document larger than memory. A spec-correct tree builder cannot run in space proportional to the open-element depth, because the adoption agency and text coalescing move nodes the walk has already emitted, so the working tree survives until the parse ends. Againstturbohtml.parsethe gain is the object graph you never build and the tree you never keep. Againsthtml.parserthe gain is speed: a counting handler oversax_parseruns a few times faster than the same handler onhtml.parser, whose tokenizer and entity handling are pure Python, and it gets the corrected tree the standard library never builds. Asaxbenchmark op tracks this againsthtml.parseron real pages, and thestdlibmigration guide gains a section mapping eachhandle_*override onto itsSaxHandlermethod.