Skip to content

✅ test(xpath): validate fn:* functions against W3C QT3 suite#587

Merged
gaborbernat merged 2 commits into
tox-dev:mainfrom
gaborbernat:test/558-xpath-qt3-conformance
Jul 7, 2026
Merged

✅ test(xpath): validate fn:* functions against W3C QT3 suite#587
gaborbernat merged 2 commits into
tox-dev:mainfrom
gaborbernat:test/558-xpath-qt3-conformance

Conversation

@gaborbernat

@gaborbernat gaborbernat commented Jul 7, 2026

Copy link
Copy Markdown
Member

Feature #558 shipped the XPath 2.0 string functions (ends-with, string-join, lower-case, upper-case, matches, replace, and the rest of the ported subset), validated so far only by turbohtml's own tests and a differential against lxml. Coverage-green is not the same as spec-correct, so this adds the authoritative check: the W3C QT3 (XQuery/XPath 3.1) test suite, the standards body's own conformance oracle, run against turbohtml's engine.

The suite is vendored as a pinned shallow submodule under tests/conformance/qt3tests, following the tests/html5lib-tests convention. tests/conformance/test_xpath_qt3_conformance.py parses the QT3 catalog and, for each fn:* case, decides whether the query lives inside turbohtml's supported grammar. 🎯 An in-scope case runs against the case's asserted result (assert-true/assert-eq/assert-string-value/error) and fails on any mismatch. Everything turbohtml omits by design is skipped or xfailed with a per-case reason carried in the parametrize id, so the signal stays on what the engine claims to support rather than being diluted by XQuery and schema machinery.

The excluded families each map to a documented boundary: XQuery-only tests (gated on the catalog's own spec dependency), xs: schema constructors and casts, XPath 2.0 sequences, ranges, and value-comparison operators, XPath 3.x maps/arrays/arrows and higher-order functions, the XSD regex dialect (turbohtml's fn:matches/fn:replace run on Python's re, so \p/\P categories, class subtraction, the q flag, and the FORX error codes fall outside it), the typed error-code taxonomy (turbohtml raises Python exceptions), and XML namespace decomposition (turbohtml is HTML-first, so local-name equals name and namespace-uri is empty outside SVG/MathML).

Of the 2631 cases across the 33 supported families, 281 run in scope and all pass, 49 xpass where turbohtml still rejects invalid input, and the remaining 2301 are recorded with a per-case reason: skipped for unsupported grammar, xfailed for the error-code taxonomy. 🔍 No engine bug surfaced: every failure traced to one of the boundaries above, which the explanation doc now records. The one adjacent finding is that parse_xml rejects a leading BOM (auction.xml), so those context cases skip as unparseable environments; that sits in the XML tree builder, not the XPath engine.

The conformance module is omitted from the coverage gate like the existing *_differential.py oracles and skips when the submodule is absent, so it never drops the free-threaded, 3.15, or Windows coverage runs. Excluded cases use skip rather than xfail(run=False) because the latter is a pytest performance pathology at this scale.

validates #558

@gaborbernat gaborbernat added the documentation Improvements or additions to documentation label Jul 7, 2026
@gaborbernat gaborbernat force-pushed the test/558-xpath-qt3-conformance branch 3 times, most recently from a7754b5 to c5711e9 Compare July 7, 2026 14:35
@codspeed-hq

codspeed-hq Bot commented Jul 7, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 83 untouched benchmarks
⏩ 18 skipped benchmarks1


Comparing gaborbernat:test/558-xpath-qt3-conformance (62fc8ad) with main (95dfb19)

Open in CodSpeed

Footnotes

  1. 18 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@gaborbernat gaborbernat force-pushed the test/558-xpath-qt3-conformance branch 5 times, most recently from cfaac3f to e3f6cff Compare July 7, 2026 19:31
Vendor the W3C QT3 (XQuery/XPath 3.1) conformance suite as a pinned
shallow submodule and drive it from tests/conformance. The harness runs
every fn:* case turbohtml's XPath engine can express -- the XPath 1.0/2.0
string, numeric, and boolean functions behind tox-dev#558 -- against the case's
asserted result, and marks the families turbohtml omits by design xfail
with a per-case, spec-justified reason.

281 in-scope cases pass; 2301 are xfailed (XQuery-only, xs: schema
types, sequences/ranges/value-comparisons, XPath 3.x, XSD-dialect regex,
the typed error-code taxonomy, XML namespace decomposition); 49 xpass
where turbohtml still rejects invalid input. No engine bug surfaced:
every deviation traces to a documented boundary.

Omitted from the coverage gate like the *_differential.py oracles; the
module skips when the submodule is absent.
@gaborbernat gaborbernat force-pushed the test/558-xpath-qt3-conformance branch from e3f6cff to 62fc8ad Compare July 7, 2026 21:46
@gaborbernat gaborbernat merged commit db0a1e6 into tox-dev:main Jul 7, 2026
49 checks passed
@gaborbernat gaborbernat deleted the test/558-xpath-qt3-conformance branch July 10, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant