✅ test: error on missing submodule instead of silent skip#586
Merged
gaborbernat merged 1 commit intoJul 7, 2026
Merged
Conversation
Conformance suites parametrize over glob("<submodule>/*.dat"). With the
html5lib-tests submodule not checked out, the glob returned nothing, so
those tests collected zero cases and pytest reported green -- hiding that
conformance never ran.
Each submodule-gated module now raises RuntimeError at import when its
vendored data is absent, naming the exact git command to fix it. CI always
inits submodules, so the guard is # pragma: no cover (on the if, to exclude
the whole block from the 100% gate).
The benchmark corpus (tools/bench-data) swaps its silent skip for a hard
error too, but only inside the test body, which runs solely under
--codspeed; a plain pytest run still deselects benchmarks untouched.
Merging this PR will not alter performance
Comparing Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The conformance suites parametrize over
glob("tests/html5lib-tests/<subdir>/*.dat"). When that submodule is not checked out the glob matches nothing, so the suites collect zero cases and pytest reports green. Conformance never ran, and nothing said so. A contributor who forgotgit submodule update --init, or any CI path that skipped submodule checkout, would ship against a suite that silently tested nothing.Every submodule-gated module now raises
RuntimeErrorat import when its vendored data is absent, naming the exact command to fix it. The guard sits on theifso a missing submodule fails collection loudly instead of degrading to an empty parametrization. CI checks out submodules (actions/checkoutwithsubmodules: true), so the guard never fires there and carries# pragma: no coveron theifline to keep the whole block out of the 100% coverage gate. This covers the three named suites plus the tree-serialize, minify round-trip, markdown, entities, and treebuilder-differential modules, which read the same corpus.The benchmark corpus under
tools/bench-datachanges the same way: its silentpytest.skipbecomes a hard error. That raise lives inside the test body, which runs only under--codspeed, so a plainpytestrun still deselects the benchmarks and is untouched. A--codspeedrun without the corpus now fails loudly rather than measuring nothing.No changelog fragment.