Skip to content

v1.6.0

Choose a tag to compare

@MattHazard MattHazard released this 05 Oct 23:06
· 20 commits to main since this release

New

  • Zarr v3 support. Reading works with the existing calls (parallelReadZarr, read_zarr, zarr(path)) for blosc (any cname and shuffle), zstd, gzip, crc32c, transpose (any axis order), little or big endian, sharding with the index at either end, both chunk key encodings, and v3 fill values (NaN, ±Infinity, bit patterns).
  • Writing Zarr v3 is opt-in: zarr_format=3 in Python, 'zarr_format', 3 in MATLAB. New v3 arrays default to C order with chunk files named c/0/0/0. Sharding uses shards= (shard shape) with chunks= (inner chunks) in Python and chunk_shape in MATLAB. Writes into an existing array keep its format, including v3 arrays made by zarr-python and TensorStore.
  • convert_to_v3 (Python) and convertZarrToV3 (MATLAB, new MEX file) turn a v2 array into v3 in place by rewriting only the metadata; .zattrs attributes move into zarr.json.
  • Arrays with any number of dimensions, from 0D up to NumPy's limit in Python, and uncompressed chunks (cname='none').
  • Reads and writes v2 arrays compressed with numcodecs zstd and zlib, and reads bool arrays (logical in MATLAB).
  • parallelWriteZarr has an 'order' option.
  • C-order reads now run as fast as F-order (previously ~2.1× slower, with twice the memory) and C-order writes are ~1.1–1.3× F-order (previously ~27× slower).
  • Python write_zarr writes C-order arrays and strided views in place instead of copying them to F order first (that copy took ~12 s per GB).
  • Crop writes into sharded arrays are ~4.3× faster (22.2 s → 5.2 s for an unaligned 2048×2048×512 region of a 32 GiB array): inner chunks outside the region are copied as stored instead of decompressed and recompressed.

Changes

  • Python read_zarr returns arrays in the file's storage order by default; pass order='F' for the previous behavior.
  • MATLAB writes 2D data as a true 2D array (shape [m n]) instead of [m n 1]. The 3D-style arguments (6-value bbox, 3-value chunks) still work on 2D arrays.
  • Missing chunks are read as the exact fill value (previously filled byte by byte, e.g., 1799 for a uint16 fill value of 7).
  • Python region writes convert the data to the existing array's type, as MATLAB does.
  • Crop writes into sharded arrays leave inner chunks outside the written region exactly as they were stored.

Fixes

  • Sparse writes no longer leave old data in chunks rewritten as all zeros, and written zeros no longer read back as a nonzero fill value.
  • bbox writes no longer lose data around the box: C-order writes overwrote existing data in partly covered chunks, and F-order writes ending inside the last partial chunk along y or z overwrote data up to the array's edge. C-order reads of a missing chunk returned 0 instead of the fill value.
  • Rewriting an existing zarr file on Windows silently kept the old data and left temporary files behind; a failed move is now an error.
  • Fixed a crash on Windows when reading or writing sharded arrays with multiple threads.
  • Sharding: crop writes read existing data with the wrong chunk shape, crop writes with uuid off truncated shards, shards with subfolders failed to write, rewrites could produce unreadable shards when their first inner chunks became empty, shard shapes that are not a multiple of the inner chunk shape lost data near the far edge, and damaged shards crashed the reader (now an error naming the shard, with the index checksum verified).
  • MATLAB parallelWriteZarr with chunk_shape on new 4D+ arrays used 1 for the shard and inner sizes past the third axis.
  • 2D arrays written by other tools crashed the reader, and 4D/5D arrays read as zeros with the wrong shape.
  • Writes into arrays made by other programs with / separators failed because the chunk folders did not exist yet.
  • Python region writes with data of another type than the array's left the array unreadable, and big-endian NumPy arrays were written byte-swapped.
  • Arrays cpp-zarr cannot read (filters, groups, float16, complex, unknown codecs) now give an error naming the reason instead of crashing or returning wrong data.

Build

  • GitHub Actions run the C++, Python, and MATLAB tests on Linux, macOS, and Windows, including arrays written by zarr-python 3 and TensorStore.
  • Fixed building with CMake 4 and the Arm Mac createZarrFile compile script's library path.
  • New compile_convertZarrToV3.m for the convertZarrToV3 MEX file.

Full Changelog: v1.5.2...v1.6.0