docs: add dev blog, update 3.3.0 release notes - #4191
Open
d-v-b wants to merge 32 commits into
Open
Conversation
* fix: byte-order handling for structured dtypes in the bytes codec The bytes codec neither byte-swapped structured-dtype fields to its configured endian on encode (numpy reports byteorder '|' for void dtypes, so the top-level byteorder comparison never detected a mismatch) nor honored its endian when decoding, silently corrupting any structured data whose field byte order differed from the stored one (e.g. virtual references to external big-endian data). Encode now detects byte-order mismatches by comparing full dtypes via newbyteorder, and decode reinterprets raw bytes in the stored byte order before converting to the data type's declared byte order, so the stored layout (codec state) and the in-memory layout (array data type) are independent. Closes zarr-developers#4141 Assisted-by: ClaudeCode:claude-fable-5 * test: fold structured byte-order cases into existing bytes codec tests Extend test_endian's parametrization with structured dtypes and test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus stored-layout and decoded-dtype assertions, instead of adding parallel test functions for the same properties. Assisted-by: ClaudeCode:claude-fable-5 * refactor: rename stored_dtype to view_dtype in BytesCodec decode The variable is the dtype used to view the raw chunk bytes (byte order from the codec's endian configuration), not a property of the stored data or of the returned buffer, which always carries the array's declared dtype. Assisted-by: ClaudeCode:claude-fable-5 * docs: note that the decode-side byte-order conversion copies the chunk Assisted-by: ClaudeCode:claude-fable-5
Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>
Configure the Material blog plugin so docs/blog/index.md renders a reverse-chronological post list instead of a blank page. The plugin ships with mkdocs-material, which is already pinned, so no new dependency is needed. Archive and category indexes are disabled -- one post doesn't justify the extra navigation. Fill in the [NEEDS INPUT] placeholders in the 3.3.0 post from the release notes and user guide: the zarr-developers#3524 regression report, the zarr-developers#3885 / zarr-developers#3925 / zarr-developers#3987 / zarr-developers#3004 pull requests, runnable config examples for both FusedCodecPipeline and the coalescing knobs, and links to the release notes and config guide. Also correct two accuracy issues: FusedCodecPipeline is opt-in and experimental (not on by default, unlike coalescing), and the sharding example now consistently says four inner chunks rather than switching to three mid-paragraph. Assisted-by: ClaudeCode:claude-opus-5
The `FusedCodecPipeline` entry advertised "up to ~24x writes / ~14x reads on many-chunks-per-shard layouts, more with compression". Those figures did not reproduce on a 10-core M4: the best measured result across the repo's own benchmark layouts was ~3.3x, and the many-chunks-per-shard layout the claim is scoped to is the one shape that gets no threading at all, because the pool fans out over outer chunks and that layout has a single outer chunk per operation. Also drop "no regressions on compute-bound workloads": zstd measured 0.4-0.5x (i.e. a regression) on single-shard layouts. Numbers that depend on hardware and layout belong in benchmark output, not in a release note, so state the improvement qualitatively instead. Assisted-by: ClaudeCode:claude-opus-5
Replace the personal gist link in the 3.3.0 blog post with two self-contained examples that ship with the docs, following the existing examples/ convention (PEP 723 header, README, docs page, nav entry). codec_pipeline_performance measures BatchedCodecPipeline against FusedCodecPipeline across three layouts, with and without compression, at one and at cpu_count workers. It reports speedups below 1.00x rather than hiding them, so the zstd single-shard regression is visible. sharding_coalescing demonstrates byte-range coalescing by counting the fetches the coalescer plans and timing them against a latency-wrapped store. Two details are needed to make the effect observable and are documented in the example: the selection must be strided (a contiguous read produces adjacent ranges that merge under any setting), and LatencyStore must not forward get_ranges to the wrapped store, or the latency is bypassed and every configuration looks identical. Both use timeit.Timer for timing rather than a hand-rolled loop, and operator.setitem/getitem via functools.partial rather than lambdas. Assisted-by: ClaudeCode:claude-opus-5
Reads more idiomatically than `Ellipsis` for a whole-array selection, and naming it `everything` makes the three call sites self-describing. Assisted-by: ClaudeCode:claude-opus-5
github-actions
Bot
added
the
needs release notes
`tests/test_examples.py` runs each example with `uv run` in an isolated environment containing only the dependencies its PEP 723 header declares. The example imported `LatencyStore` from `zarr.testing.store`, which hard-imports `pytest` at module level, so the script died with ModuleNotFoundError and failed CI on py3.12-min_deps and macOS py3.14. Define the latency wrapper in the example instead, subclassing the public `WrapperStore`. This keeps the script runnable with only zarr and numpy, and makes the mechanism it depends on explicit: `get` sleeps, and `get_ranges` is inherited from the `Store` ABC rather than `WrapperStore`'s forwarding version, so each merged fetch pays the latency once instead of bypassing it. `_with_store` is overridden because `WrapperStore` rebuilds the wrapper when opening read-only and would otherwise drop the latency setting. Assisted-by: ClaudeCode:claude-opus-5
The contributing guide asks that most pull requests carry a news fragment, and the self-merge policy lists one among the standard merge requirements. The CI check only validates the filename structure of fragments that exist, so it passed without one. Assisted-by: ClaudeCode:claude-opus-5
github-actions
Bot
removed
the
needs release notes
d-v-b
marked this pull request as ready for review
July 28, 2026 13:14
The example only measured zstd level 0, the fast Zarr v3 default, so it never covered the slow-encoder case. Add gzip level 5 as a third compressor: it moves the bottleneck from async scheduling to codec compute, which is a different regime for the two pipelines and worth showing side by side. Also drop the claim in the notes that threading "can pay off" with compression. Measured on a 10-core M4, the worker count barely moves any compressed row -- the pool parallelizes over outer chunks, and adding compression does not change how many of those there are. Assisted-by: ClaudeCode:claude-opus-5
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4191 +/- ##
=======================================
Coverage 93.88% 93.88%
=======================================
Files 91 91
Lines 12622 12622
=======================================
Hits 11850 11850
Misses 772 772 🚀 New features to boost your workflow:
|
Co-authored-by: Kyle Barron <kylebarron2@gmail.com>
Co-authored-by: Kyle Barron <kylebarron2@gmail.com>
Co-authored-by: Kyle Barron <kylebarron2@gmail.com>
Co-authored-by: Kyle Barron <kylebarron2@gmail.com>
…rkload The example measured a 1D array with 100k-element chunks and reported that the thread pool never mattered. That contradicted the blog post, and the example was wrong on two counts. Workload: switch to the workload the post's numbers actually came from -- 4096x4096 int32 in 16 shards of 1024x1024, each holding 256 inner chunks of 64x64, over both MemoryStore and LocalStore. The 64x64 chunks are the load-bearing detail: small enough that per-chunk scheduling dominates uncompressed IO, large enough that per-chunk gzip is real work to spread across cores. The previous 100k-element chunks were far too coarse for a thread pool to have anything to do. Each regime also gets the data that exercises it -- arange for uncompressed, noisy integers for gzip, which would otherwise compress away to nothing. Measurement: run the whole operation inside `zarr.config.set`, not just the array construction. The pipeline class is resolved at construction, but `codec_pipeline.max_workers` is read per operation, so timing calls made outside the config block silently used the ambient worker count and made every configuration look identical. With both fixed, the example reproduces the documented behavior: gzip writes at 1 worker are slower than the default and a thread pool recovers 1.4-1.8x, while uncompressed writes are fastest at 1 worker and lose ~0.6x to pool overhead. Assisted-by: ClaudeCode:claude-opus-5
Two suggestions from the review on #246 were not yet applied: - Replace the `4 << 20` bit-shift literals with `4 * 1024**2`, which reads more clearly as bytes -> KiB -> MiB. The suggestion wrote `1024 ** 3`, but these values are MiB, so the exponent is 2. - Reword the closing paragraph to state the compiled-code plan directly instead of framing it as something we "should look to". The rest of the review (subchunks phrasing, dropping the `N`/`M` notation, the "first step of a larger direction" opener, and the author description) is already in the branch. Assisted-by: ClaudeCode:claude-opus-5
Merge upstream/main (zarr-developers#4146 open_like, zarr-developers#4119 zarr-metadata model layer) and refresh this PR's news fragment so it describes what the branch actually contains now: - The pipeline example no longer sweeps "several array layouts"; it compares the two pipelines on one sharded array across two stores and two codec regimes, which is what shows the thread-pool tradeoff. - Mention the removal of the hardware-specific speedup figures from the FusedCodecPipeline release note, which is part of this PR's diff but went undescribed. Both incoming PRs carried their own fragments (changes/3352.bugfix.md and packages/zarr-metadata/changes/4119.*), so no other entry was needed. Assisted-by: ClaudeCode:claude-opus-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
If merged, I would tag the associated commit with the 3.3.0 tag and cut the release.
For reviewers
Have a look a the blog content, let me know if you want changes.
Author attestation
TODO
docs/user-guide/*.mdchanges/