ci(wpt): fail-fast on missing corpus + dedicated CI fetcher with shared cache#2
Merged
Merged
Conversation
…hared cache The matrix-with-wheel-install run combined to 82.1% coverage (vs 96.7% locally on the same suite) because ~470 conformance tests were silently no-op-ing: ``reference/wpt/`` is gitignored and not in the sdist, so matrix jobs never saw the corpus and ``pytest.skip()`` / ``return []`` swallowed all the test_wpt*.py parametrize. The CI was effectively only running the 110 non-conformance tests. Fix has three parts: * **Tests fail-fast** when the corpus is absent. ``conftest.py``, ``test_wpt_compare.py``, and ``test_wpt_generate.py`` now raise ``FileNotFoundError`` with an actionable message instead of silently collecting zero cases. A missing fixture is a CI / dev-env error, not a runtime condition the tests should tolerate. * **New ``scripts/fetch_wpt_corpus.sh``** — security-hardened, dedicated to the CI use case. Pinned ``WPT_REF`` SHA; HTTPS-only sparse-checkout of just ``urlpattern/`` + ``resources/``; ``--filter=blob:none`` so unrelated blobs never land on the runner; post-fetch ``git rev-parse HEAD`` verify against the pin; 10 MiB cap on every JSON fixture before ``json.loads`` (defense against a parser-DoS surface); top-level shape check (must be a list of objects/strings); ``--verify`` mode for re-checking a restored cache. ``umask 022`` and a hardcoded ``PATH`` defend against environment manipulation. * **``ci.yml`` ``wpt-corpus`` job**. Caches by content of the fetch script (bumping ``WPT_REF`` rotates the key), uploads the corpus as an artifact, and matrix jobs download it once per workflow run. Both ``test-stable`` (9 shards) and ``test-prospective`` consume the artifact. ``ci-ok`` (the aggregate ``CI green`` status check) lists the new job in ``needs`` so the merge gate fails closed if the fetch job breaks. The dev-side ``scripts/fetch_references.sh`` retains its own ``WPT_REF`` duplicate — a comment in the new script flags the lockstep requirement.
2 tasks
chad-loder
added a commit
that referenced
this pull request
May 12, 2026
#3) PR #2 fixed the silent-skip of ~470 conformance tests in the matrix-with- wheel-install path; CI's combined coverage climbed from 82.1% to 96.7%. With the matrix now exercising the full suite, the 82 ratchet has no signal value. Move it to 95 — ~1.7-point headroom below the current run absorbs cross-OS / cross-Python variation without permitting silent regressions. The target stated in the previous ratchet comment was 90% sustained; we're now well above that, so the ratchet itself becomes the contract. Future tests should keep coverage at or above 95. Co-authored-by: chad-loder <26261238+chad-loder@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI's matrix-with-wheel-install reported 82.1% coverage vs 96.7% locally on the same 580-test suite. Root cause:
reference/wpt/is gitignored and not packed into the sdist, so all 469 WPT-corpus tests silently no-op'd in matrix jobs (pytest.skipin conftest,return []in the data-driven test_wpt_*.py files).Three changes:
Fail-fast tests.
conftest.py,test_wpt_compare.py,test_wpt_generate.pynowraise FileNotFoundErrorinstead of silently producing an empty parametrize. A missing corpus is now a loud CI / dev-env error.New
scripts/fetch_wpt_corpus.sh— security-hardened, dedicated to CI. Pinned SHA, HTTPS sparse-clone of justurlpattern/+resources/, post-fetch SHA verify against the pin, 10 MiB cap on every JSON fixture beforejson.loads, top-level shape check,--verifymode for re-checking restored caches.umask 022and hardcodedPATH.wpt-corpusjob inci.yml.actions/cachekeyed on the fetch script's content (bumpingWPT_REFrotates the key automatically), uploads as an artifact,test-stable+test-prospectivedownload once per workflow run. Listed inci-ok'sneedsso the merge gate fails closed on fetch breakage.Expected result
If the path-mapping theory is correct, CI's combined coverage should jump from 82.1% to ~96.7% (matching local). The ratchet at 82% stays — separate PR to bump after we confirm the number on green main.
Test plan
scripts/fetch_wpt_corpus.sh --verifypasses against the local symlinked corpusjust test-covreports 96.7% (1300 stmts, 580 passed, 19 skipped)actionlint .github/workflows/ci.ymlcleanshellcheck scripts/fetch_wpt_corpus.shcleanwpt-corpusjob populates the cache + uploads the artifacttest-stablematrix jobs collect all 580 tests (vs ~190 currently)coverageaggregate clears the 82% floor (target: ~96%)CI greenaggregate goes green so the ruleset allows merge