⚡ perf(core): shrink tokenizer and DOM state#640
Merged
Conversation
Large flat documents paid for attribute data consumers did not use. DOM construction also repeated allocations for tags and per-node caches. Add an opt-in tag-and-text tokenizer path and compact shared storage. Preserve the default token contract while reducing parser allocations.
gaborbernat
added
enhancement
for more information, see https://pre-commit.ci
Linux C11 builds did not expose SSIZE_MAX through PyMem_New, and clang-tidy rejected the tagged text span's integer-to-pointer cast. Allocate the fixed-size caches by byte count and copy the tagged value into the pointer field.
Merging this PR will improve performance by 8.79%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | test_feature[tables-spec] |
443.2 µs | 388.3 µs | +14.14% |
| ⚡ | test_feature[set-text] |
539.2 µs | 482.3 µs | +11.79% |
| ⚡ | test_feature[set-html] |
562.9 µs | 505.9 µs | +11.27% |
| ⚡ | test_feature[edit] |
661.7 µs | 601.9 µs | +9.94% |
| ⚡ | test_feature[select-has] |
216.4 µs | 197.4 µs | +9.66% |
| ⚡ | test_feature[links-rewrite] |
736.9 µs | 673.7 µs | +9.38% |
| ⚡ | test_feature[chain] |
670.5 µs | 614.9 µs | +9.04% |
| ⚡ | test_feature[xpath] |
716.3 µs | 660 µs | +8.53% |
| ⚡ | test_feature[conformance] |
1.2 ms | 1.1 ms | +6.02% |
| ⚡ | test_feature[links-extract] |
1.3 ms | 1.2 ms | +5.59% |
| ⚡ | test_feature[observe] |
1,031.3 µs | 978.4 µs | +5.41% |
| ⚡ | test_feature[treewalk] |
1.1 ms | 1.1 ms | +5.08% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing gaborbernat:simple-parser (329eb40) with main (ecd53e1)
Footnotes
-
18 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streaming consumers that only need tag names and text still pay to allocate attribute data. DOM parsing duplicates known tag names and allocates unused query storage.
capture_attributes=Falseremoves the first cost. Compact text nodes and shared tag storage reduce the rest; attribute metadata and source spans use packed allocations, while query caches materialize on demand. With a 43.8 MB PyPI index snapshot, anchor extraction improves 7.2% and DOM parsing improves 8.8%; traced peak memory falls 14.0% and peak RSS falls 11.2%. Generated documentation tables record the changes across existing build and serialization cases.Paired PGO runs cover every path touched by the patch, from builders and parsing through lossless output and XSLT. The percentages above come from quiet runs. Later host contention produced high-variance rows, which do not inform the comparison.