Skip to content

✅ test: error on missing submodule instead of silent skip#586

Merged
gaborbernat merged 1 commit into
tox-dev:mainfrom
gaborbernat:test/harden-submodule-gated-tests
Jul 7, 2026
Merged

✅ test: error on missing submodule instead of silent skip#586
gaborbernat merged 1 commit into
tox-dev:mainfrom
gaborbernat:test/harden-submodule-gated-tests

Conversation

@gaborbernat

Copy link
Copy Markdown
Member

The conformance suites parametrize over glob("tests/html5lib-tests/<subdir>/*.dat"). When that submodule is not checked out the glob matches nothing, so the suites collect zero cases and pytest reports green. Conformance never ran, and nothing said so. A contributor who forgot git submodule update --init, or any CI path that skipped submodule checkout, would ship against a suite that silently tested nothing.

Every submodule-gated module now raises RuntimeError at import when its vendored data is absent, naming the exact command to fix it. The guard sits on the if so a missing submodule fails collection loudly instead of degrading to an empty parametrization. CI checks out submodules (actions/checkout with submodules: true), so the guard never fires there and carries # pragma: no cover on the if line to keep the whole block out of the 100% coverage gate. This covers the three named suites plus the tree-serialize, minify round-trip, markdown, entities, and treebuilder-differential modules, which read the same corpus.

The benchmark corpus under tools/bench-data changes the same way: its silent pytest.skip becomes a hard error. That raise lives inside the test body, which runs only under --codspeed, so a plain pytest run still deselects the benchmarks and is untouched. A --codspeed run without the corpus now fails loudly rather than measuring nothing.

No changelog fragment.

Conformance suites parametrize over glob("<submodule>/*.dat"). With the
html5lib-tests submodule not checked out, the glob returned nothing, so
those tests collected zero cases and pytest reported green -- hiding that
conformance never ran.

Each submodule-gated module now raises RuntimeError at import when its
vendored data is absent, naming the exact git command to fix it. CI always
inits submodules, so the guard is # pragma: no cover (on the if, to exclude
the whole block from the 100% gate).

The benchmark corpus (tools/bench-data) swaps its silent skip for a hard
error too, but only inside the test body, which runs solely under
--codspeed; a plain pytest run still deselects benchmarks untouched.
@gaborbernat gaborbernat added the bug Something isn't working label Jul 7, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 7, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 81 untouched benchmarks
⏩ 18 skipped benchmarks1


Comparing gaborbernat:test/harden-submodule-gated-tests (ba276fc) with main (dee9eae)

Open in CodSpeed

Footnotes

  1. 18 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@gaborbernat gaborbernat merged commit 8349b19 into tox-dev:main Jul 7, 2026
48 checks passed
@gaborbernat gaborbernat deleted the test/harden-submodule-gated-tests branch July 10, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant