Add docs-tests.yml; remove test_doc_snippets.py from rust-test.yml#399
Merged
Add docs-tests.yml; remove test_doc_snippets.py from rust-test.yml#399
Conversation
…est.yml PR #389 (Batch A) shipped six broken HAD doc snippets in 2026-04 that no CI run caught because rust-test.yml only triggers on rust/, diff_diff/, tests/, pyproject.toml, and the workflow file — none of which include docs/. PR #396 patched the snippets but did not address the structural gap. This PR addresses it. Two changes: 1. New .github/workflows/docs-tests.yml — separate workflow that runs `pytest tests/test_doc_snippets.py -v` on a single ubuntu-latest / py3.14 / pure-Python runner. Triggers on docs/, diff_diff/, tests/test_doc_snippets.py, pyproject.toml, and the workflow file itself; same ready-for-ci label gate as rust-test.yml / notebooks.yml. Mirrors notebooks.yml's shape (the existing precedent for `pytest`-validated docs assets) so the two doc-validation workflows look like siblings. 2. .github/workflows/rust-test.yml: add --ignore=tests/test_doc_snippets.py to all three pytest invocations so doc snippets stop riding the code workflow. The Pure Python Fallback edit (line 193) is the only one that changes CI signal: that job runs from the repo root and was the ONLY place where test_doc_snippets.py actually executed. The two Rust-matrix edits (lines 158, 165) are defensive consistency: the matrix copies tests/ to /tmp/tests (rust-test.yml:138, 142) without docs/, so DOCS_DIR resolves to /tmp/docs/ which doesn't exist; the test collector silently skips every RST file via the guard at tests/test_doc_snippets.py:129. Adding --ignore there prevents the no-op from becoming a real run if anyone later adds `cp -r docs ...` to the copy steps. Each invocation now carries an in-YAML comment documenting which case it's the defensive vs behavior-changing edit. Verification: - python -c "import yaml; yaml.safe_load(open('.github/workflows/ docs-tests.yml')); yaml.safe_load(open('.github/workflows/ rust-test.yml'))" — both files well-formed. - pytest tests/ --ignore=tests/test_doc_snippets.py --ignore=tests/test_rust_backend.py --collect-only — 0 occurrences of test_doc_snippets in the collected set (was 115 cases collected when not ignored), confirming pytest accepts repeated --ignore flags as the existing line-193 pattern with --ignore=tests/ test_rust_backend.py already showed. After this PR opens, the workflow file itself triggers docs-tests.yml on its own change, providing the first end-to-end CI validation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Overall Assessment ✅ Looks good — no unmitigated P0/P1 findings. Executive Summary
Methodology
Code Quality
Performance
Maintainability
Tech Debt
Security
Documentation/Tests
|
P3 (Performance): rust-test.yml's `tests/**` trigger still fans out into the full Rust matrix on harness-only edits to tests/test_doc_snippets.py, even though every pytest invocation in that workflow now ignores the file. Added the GitHub Actions negative-pattern `!tests/test_doc_snippets.py` to both the push and pull_request paths blocks so a snippet-test-only edit no longer spins up the matrix. The file is owned exclusively by docs-tests.yml which still triggers (since its own `paths:` lists the file explicitly). P3 (Tech Debt): docs-tests.yml repeats the broad `labeled` / `unlabeled` trigger pattern that TODO.md:123 already tracks as avoidable CI churn for rust-test.yml + notebooks.yml under PR #269. Rather than diverge by pre-fixing the new workflow alone, extended the tracking row to include `docs-tests.yml` so the planned cross-cutting fix covers all three uniformly. Verification: python -c "import yaml; yaml.safe_load(open( '.github/workflows/rust-test.yml'))" succeeds; em-dash sweep clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owner
Author
|
/ai-review |
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall Assessment ✅ Looks good — no unmitigated P0/P1 findings. Executive Summary
Methodology
Code Quality
Performance
Maintainability
Tech Debt
Security
Documentation/Tests
|
P2: docs-tests.yml only path-matched tests/test_doc_snippets.py, but pytest auto-loads tests/conftest.py for any test under tests/ — and that file mutates sys.path (line 14: prepends tests/helpers/) and sets MPLBACKEND=Agg (line 18) before any test imports it. A change to tests/conftest.py that breaks either of those load-time steps would silently break the snippet test on every doc PR until a docs-tests.yml-triggering file happened to change alongside it, leaving a real CI blind spot. Fix: added 'tests/conftest.py' to both the push and pull_request `paths:` blocks in docs-tests.yml. An inline comment on the push block explains the dependency; the pull_request block doesn't repeat the prose to avoid drift between the two copies. Verified the workflow YAML is well-formed via yaml.safe_load. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owner
Author
|
/ai-review |
|
🔁 AI review rerun (requested by @igerber) Head SHA: Overall Assessment ✅ Looks good — no unmitigated P0/P1 findings. Executive Summary
Methodology
Code Quality
Performance
Maintainability
Tech Debt
Security
Documentation/Tests
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.github/workflows/docs-tests.yml— a separate workflow that ownstests/test_doc_snippets.pyand runs ondocs/**,diff_diff/**,tests/test_doc_snippets.py,pyproject.toml, and the workflow filetests/test_doc_snippets.pyfrom.github/workflows/rust-test.ymlvia--ignoreon all three pytest invocations so doc-snippet tests stop riding the code workflowWhy
PR #389 (Batch A) shipped six broken HAD doc snippets in 2026-04 that no CI run caught because
rust-test.ymlonly triggers onrust/**,diff_diff/**,tests/**,pyproject.toml, and its own workflow file — none of which includedocs/. PR #396 patched the snippets but did not address the structural gap. This PR addresses it by mirroring the precedent already set bynotebooks.yml(separate workflow scoped todocs/tutorials/**+diff_diff/**for notebook validation).Trigger behavior
rust-test.yml?docs-tests.yml?diff_diff/**onlydocs/**onlytests/test_doc_snippets.pyonly--ignoreskips it)pyproject.tomlSubtlety on the
rust-test.ymleditsLines 158/165 (the Rust matrix) are defensive consistency edits, not behavior changes. The matrix copies
tests/to/tmp/tests(rust-test.yml:138, 142) withoutdocs/, sotests/test_doc_snippets.py:23DOCS_DIR = Path(__file__).resolve().parent.parent / "docs"resolves to/tmp/docs/which doesn't exist; the collector silently skips every RST file via the guard attests/test_doc_snippets.py:129. Adding--ignorethere prevents the no-op from becoming a real run if anyone later addscp -r docs ...to the copy steps.Line 193 (Pure Python Fallback) is the only edit that changes CI signal — that job runs from the repo root and was the only place the snippet test actually executed. Each invocation now carries an in-YAML comment documenting which case applies.
Methodology references
Validation
python -c "import yaml; yaml.safe_load(open('.github/workflows/docs-tests.yml')); yaml.safe_load(open('.github/workflows/rust-test.yml'))"— both files well-formedpytest tests/ --ignore=tests/test_doc_snippets.py --ignore=tests/test_rust_backend.py --collect-onlyshows 0 occurrences oftest_doc_snippetsin the collected set (was 115 cases when not ignored), confirming the new--ignorecomposes with the existing--ignore=tests/test_rust_backend.pySecurity / privacy