docs: test more code snippets in docs#4018
Open
d-v-b wants to merge 28 commits into
Open
Conversation
…evelopers#4016) Design for fixing issue zarr-developers#4016 (invalid create_array(mode="w") in docs) and preventing recurrence: per-case remediation of the 12 non-executed python blocks (S3 via moto, config blocks via exec="true", GPU via the gpu marker, explicit opt-out for non-Python blocks) plus a guard test asserting every docs python block is either executed or explicitly opted out with a reason. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…spec Replace the bespoke hidden-moto-setup-block approach for the S3 example with a marker-bound model: a block declares markers="s3"/"gpu" on the fence, and the harness binds each marker to the infra/env it needs (s3 -> moto fixture in the default doctest env; gpu -> gputest env via pytest -m gpu). Symmetric declaration; the asymmetry is only in what each marker resolves to. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…4016, zarr-developers#4017) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Result: A-env-var
A bare zarr.create_array("s3://bucket/key", ...) with NO storage_options
reaches a moto server when AWS_ENDPOINT_URL is set process-wide. s3fs/
aiobotocore honor the env var, so the visible docs block can stay clean.
Caveat: moto[s3,server] currently lives only in the 'remote-tests'
dependency group, not 'test'; the doctest hatch env (dependency-groups=
['test']) does NOT have moto installed. The downstream real-fixture task
must add moto[s3,server] (and requests) to the doctest env extras.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rr-developers#4016) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r mechanism) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rr-developers#4017) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…break the build markdown-exec's `exec="true"` means "run at build to render output"; build runners have no GPU/cupy and no moto server, so tagging the GPU/S3 examples exec="true" made `mkdocs build --strict` abort. Introduce a separate `test="true"` flag that our tests/test_docs.py harness keys on (markdown-exec ignores it): a block is validated if exec="true" OR test="true". The GPU and S3 examples become test="true" (+markers) and are no longer run at build. Also: a test="true"-only python fence placed before an exec="true" block of the same page disrupts markdown-exec's build execution of the later block (the quickstart ZipStore example failed with FileNotFoundError). Move the S3 example to the end of quick-start.md so no shared-session exec block follows it; document the constraint in the guard docstring and the design spec. Verified: full docs test suite green (57 passed, 2 skipped), `mkdocs build --strict` exits 0, prek (ruff/mypy/...) clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…, zarr-developers#4017) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address roborev branch-review findings (job 186):
- Medium: moto-api reset POST ran before server.stop()/env restore in the
finally block; if it raised, the fixed-port server thread leaked and
AWS_ENDPOINT_URL was left dangling. Nest the reset in its own try/finally so
server.stop() and env restoration always run.
- Low: f"{S3_ENDPOINT}/moto-api/reset" double-slashed (constant ends in /);
drop the extra slash.
- Low: AWS_SECRET_ACCESS_KEY/AWS_ACCESS_KEY_ID were setdefault'd but never
restored; save/restore all three mutated env vars uniformly.
- Low: cli.md create_array lacked overwrite=True, non-idempotent across local
runs; add it.
Verified: full docs suite green (57 passed, 2 skipped), s3+cli pass on repeated
runs, mkdocs build --strict exits 0, prek clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… scope
Address roborev branch-review findings (job 188):
- Medium: the gpu docs example ran in NO environment. The gputest env lacks
pytest-examples, so test_docs.py's module-level importorskip("pytest_examples")
skipped the whole module under `pytest -m gpu` -- the gpu case was never
collected even on GPU hardware, yet the guard reported it "validated" via
test="true". Add pytest-examples to the gputest env; confirmed gpu-demo now
collects under `hatch -e gputest run pytest -m gpu --co`.
- Low: _session_params (collection) didn't exclude docs/superpowers/ while the
guard did -- an asymmetry that could run a cache-doc block as a real test
without the guard flagging it. Extract a shared _is_published_docs() helper
used by both, so collection and guard agree on scope.
Verified: doctest suite green (57 passed, 2 skipped), gpu-demo collectable in
gputest, mkdocs build --strict exits 0, prek clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…trip Address roborev branch-review findings (job 190): - Low: the "test=true-only block must not precede an exec=true block on the same page" constraint was documented but unenforced (only caught by mkdocs --strict). Add test_test_only_blocks_come_last, which fails fast/locally when a test-only block has a smaller start_line than a later same-file exec=true block. Negative- checked: it catches an exec block added after the gpu test-only block. - Low: _markers_for built pytest.MarkDecorator objects that _session_params immediately reduced to .name and rebuilt; replace with _marker_names returning raw strings, building decorators once at param time. Verified: doctest suite green (58 passed, 2 skipped), placement test passes and fails on a planted violation, prek clean, mkdocs build --strict exits 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address roborev branch-review finding (job 192): the placement guard's docstring claimed any non-exec="true" python fence before an exec="true" block breaks the build, but the guard only checked test="true" blocks, leaving exec="false"-before- exec="true" arrangements (data_types.md, performance.md) unguarded. Investigation (experiments + markdown-exec's SuperFences integration) established the real mechanism: a non-executed python fence (test="true" OR exec="false") before an exec="true" block disrupts build-time execution of a later *state-dependent* block (needs a cross-block dependency to surface, which is why the standalone exec="true" blocks under those exec="false" opt-outs build fine). CI ran non-strict `mkdocs build`, so such a failure would have merged as a silent warning. - ci/docs.yml: `mkdocs build` -> `mkdocs build --strict` so any build-time exec failure (including the exec="false" case) fails CI authoritatively. Verified: a clean build currently emits zero warnings, so --strict passes today. - Narrow the placement guard's docstring and the spec to the real (state-dependent) mechanism, and frame test_test_only_blocks_come_last as a conservative fast-feedback convention with --strict as the authoritative check -- no longer over-claiming. Verified: docs suite green (58 passed, 2 skipped), `mkdocs build --strict` exits 0, prek clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spec and implementation plan under docs/superpowers/ were working artifacts, not published documentation (they were never in the mkdocs nav). The spec is preserved in a public gist; the plan is a local execution record. Remove both from the repo and drop the now-stale spec path from the test_docs.py module docstring. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The _is_published_docs helper existed only to skip docs/superpowers/ design-doc caches; those files are no longer in the repo, so the helper and its three call sites were dead code referencing a nonexistent directory. Remove them — every find_examples(DOCS_ROOT) result is now published docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both test_store/test_fsspec.py and test_docs.py stood up their own moto ThreadedMotoServer (ports 5555 and 5556). Extract a single session-scoped `moto_server` fixture + MOTO_ENDPOINT_URL constant into tests/conftest.py and have both consumers reuse it: - test_fsspec.py: s3_base now returns the shared moto_server; its per-test `s3` fixture (bucket "test", explicit endpoint, event-loop cleanup) is unchanged. - test_docs.py: docs_s3_backend depends on moto_server and adds only its docs-specific layer (process-wide AWS_ENDPOINT_URL + "example-bucket"); it no longer owns the server lifecycle. One server now serves the whole session; each consumer creates and moto-api-resets its own bucket. Verified: test_fsspec.py (96 passed), the docs suite (58 passed), both together in one session (154 passed), and the full standard suite in the optional env (5956 passed); prek clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The contributing guide explained exec="true" but said nothing about test="true", the exec="false"+reason opt-out, the guard that requires one of them, or the placement constraint -- so a contributor could write a bare block and hit the guard with no explanation. Add a "Validating code blocks: exec vs test" section covering: - exec="true" (build-render) vs test="true" (validate-only) and when to use each - the exec="false" reason="..." opt-out and the test_no_unvalidated_blocks guard - markers="gpu"/"s3" for infra-bound blocks - the placement rule (test-only blocks come last) + --strict CI Attribute examples are shown as inline code rather than nested ```python fences, so pytest-examples' find_examples never mistakes the teaching examples for real blocks (verified: only the two genuine blocks in contributing.md are collected). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…into docs-cleanups
d-v-b
commented
May 29, 2026
| print("Hello world") | ||
| ``` | ||
|
|
||
| #### Validating code blocks: `exec` vs `test` |
Contributor
Author
There was a problem hiding this comment.
this addition to the docs explains the logic added in this PR
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4018 +/- ##
=======================================
Coverage 93.55% 93.55%
=======================================
Files 88 88
Lines 11896 11896
=======================================
Hits 11129 11129
Misses 767 767 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed an invalid
zarr.create_arrayexample in the quick-start documentation (it passed an unsupportedmodeargument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested.The changes here touch on our s3 tests because we needed to re-use the mock s3 backend to ensure that the s3 example in the docs runs properly.
closes #4017 and #4016