Skip to content

✅ test(dom): validate source locations against parse5#584

Merged
gaborbernat merged 3 commits into
tox-dev:mainfrom
gaborbernat:test/548-parse5-location-conformance
Jul 7, 2026
Merged

✅ test(dom): validate source locations against parse5#584
gaborbernat merged 3 commits into
tox-dev:mainfrom
gaborbernat:test/548-parse5-location-conformance

Conversation

@gaborbernat

Copy link
Copy Markdown
Member

parse(source_locations=True) ships as turbohtml's take on parse5's sourceCodeLocationInfo, so the spans it stamps only mean something if they land where parse5's do. 🎯 This validates them against the reference implementation on parse5's own location-info corpus: the five real-world documents (CERN, a DX portal, the parse5 GitHub page, the WHATWG HTML spec, and a MediaWiki article) that parse5's suite parses to exercise its location code.

parse5 is JavaScript, so a committed Node runner (tools/bench/node/parse5_location_runner.js, reusing the existing parse5 competitor setup) emits every element's start-tag, end-tag, and per-attribute spans, and a pytest harness parses the same newline-normalized input through turbohtml and compares span-for-span. Both sides key elements by start-tag start offset and normalize to turbohtml's convention (parse5's columns are 1-based, its offsets and lines match), and turbohtml parses with scripting=True to line the trees up with parse5's default. The corpus is a pinned shallow submodule under tests/conformance/parse5; the harness skips when Node or the submodule is absent, the way the *_differential.py oracles do, so it stays out of the coverage gate. All 9767 elements agree on all three span kinds.

The corpus surfaced a real bug. 🐛 A body, html, or form element that the source closes with its own tag left source_location.end_tag at None: these three close through an insertion-mode switch or an out-of-order stack removal, never the normal pop that stamps the end-tag span, so an explicit </body>, </html>, or </form> went unrecorded. A record_end_tag_location helper now stamps them from those handlers; the start-tag and per-attribute spans were already byte-exact against parse5 before the fix.

Validates #548.

@gaborbernat gaborbernat added bug Something isn't working area:parser Tokenizer / tree construction labels Jul 7, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 7, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 83 untouched benchmarks
⏩ 18 skipped benchmarks1


Comparing gaborbernat:test/548-parse5-location-conformance (d156093) with main (4ab5e35)

Open in CodSpeed

Footnotes

  1. 18 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@gaborbernat gaborbernat force-pushed the test/548-parse5-location-conformance branch 7 times, most recently from c4c0a98 to e1274d8 Compare July 7, 2026 22:33
The conformance job checks out submodules recursively, so parse5 pulls
its own nested html5lib-tests submodule, whose encoding fixtures hold
non-UTF-8 bytes. pytest tried to collect them and aborted with a
UnicodeDecodeError. Ignore the vendored parse5 directory the same way the
unicodetools data submodule is already ignored, and drop the changelog
fragment (conformance bug fixes carry none).
@gaborbernat gaborbernat force-pushed the test/548-parse5-location-conformance branch from e1274d8 to d156093 Compare July 7, 2026 23:29
@gaborbernat gaborbernat merged commit a389e96 into tox-dev:main Jul 7, 2026
49 checks passed
@gaborbernat gaborbernat deleted the test/548-parse5-location-conformance branch July 10, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:parser Tokenizer / tree construction bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant