✅ test(xpath): validate fn:* functions against W3C QT3 suite#587
Merged
gaborbernat merged 2 commits intoJul 7, 2026
Merged
Conversation
a7754b5 to
c5711e9
Compare
Merging this PR will not alter performance
Comparing Footnotes
|
cfaac3f to
e3f6cff
Compare
Vendor the W3C QT3 (XQuery/XPath 3.1) conformance suite as a pinned shallow submodule and drive it from tests/conformance. The harness runs every fn:* case turbohtml's XPath engine can express -- the XPath 1.0/2.0 string, numeric, and boolean functions behind tox-dev#558 -- against the case's asserted result, and marks the families turbohtml omits by design xfail with a per-case, spec-justified reason. 281 in-scope cases pass; 2301 are xfailed (XQuery-only, xs: schema types, sequences/ranges/value-comparisons, XPath 3.x, XSD-dialect regex, the typed error-code taxonomy, XML namespace decomposition); 49 xpass where turbohtml still rejects invalid input. No engine bug surfaced: every deviation traces to a documented boundary. Omitted from the coverage gate like the *_differential.py oracles; the module skips when the submodule is absent.
e3f6cff to
62fc8ad
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature #558 shipped the XPath 2.0 string functions (
ends-with,string-join,lower-case,upper-case,matches,replace, and the rest of the ported subset), validated so far only by turbohtml's own tests and a differential against lxml. Coverage-green is not the same as spec-correct, so this adds the authoritative check: the W3C QT3 (XQuery/XPath 3.1) test suite, the standards body's own conformance oracle, run against turbohtml's engine.The suite is vendored as a pinned shallow submodule under
tests/conformance/qt3tests, following thetests/html5lib-testsconvention.tests/conformance/test_xpath_qt3_conformance.pyparses the QT3 catalog and, for eachfn:*case, decides whether the query lives inside turbohtml's supported grammar. 🎯 An in-scope case runs against the case's asserted result (assert-true/assert-eq/assert-string-value/error) and fails on any mismatch. Everything turbohtml omits by design is skipped or xfailed with a per-case reason carried in the parametrize id, so the signal stays on what the engine claims to support rather than being diluted by XQuery and schema machinery.The excluded families each map to a documented boundary: XQuery-only tests (gated on the catalog's own
specdependency),xs:schema constructors and casts, XPath 2.0 sequences, ranges, and value-comparison operators, XPath 3.x maps/arrays/arrows and higher-order functions, the XSD regex dialect (turbohtml'sfn:matches/fn:replacerun on Python'sre, so\p/\Pcategories, class subtraction, theqflag, and theFORXerror codes fall outside it), the typed error-code taxonomy (turbohtml raises Python exceptions), and XML namespace decomposition (turbohtml is HTML-first, solocal-nameequalsnameandnamespace-uriis empty outside SVG/MathML).Of the 2631 cases across the 33 supported families, 281 run in scope and all pass, 49 xpass where turbohtml still rejects invalid input, and the remaining 2301 are recorded with a per-case reason: skipped for unsupported grammar, xfailed for the error-code taxonomy. 🔍 No engine bug surfaced: every failure traced to one of the boundaries above, which the explanation doc now records. The one adjacent finding is that
parse_xmlrejects a leading BOM (auction.xml), so those context cases skip as unparseable environments; that sits in the XML tree builder, not the XPath engine.The conformance module is omitted from the coverage gate like the existing
*_differential.pyoracles and skips when the submodule is absent, so it never drops the free-threaded, 3.15, or Windows coverage runs. Excluded cases useskiprather thanxfail(run=False)because the latter is a pytest performance pathology at this scale.validates #558