Skip to content

Release 4.13.1

Latest

Choose a tag to compare

@FrancescAlted FrancescAlted released this 15 Sep 10:32
· 154 commits to main since this release

Changes from 4.13.0 to 4.13.1

A maintenance and performance follow-up to 4.13.0 focused on remote data
access and expanded platform support. Lazy access is now the default for remote
arrays and container datasets, local HDF5 files are read directly via h5py
without auxiliary dependencies, warm opens replay cached bootstrap metadata
to eliminate redundant network discovery, disk caches use human-readable
directory structures, and native Windows ARM64 wheels are now built and tested.

Improvements

Remote access

  • Lazy access is the default for remote arrays and datasets. blosc2.open()
    on remote .b2nd files or container datasets (HDF5, Zarr, B2Z) now returns a
    lazy RemoteArray without requiring an explicit lazy=True. Dataset paths
    require lazy access and reject lazy=False with NotImplementedError rather
    than attempting a silent eager download. For standalone array files (such as
    .b2nd), cache_dir= keeps the default lazy access and persists fetched chunks
    in a RemoteArray carrier. Pass lazy=False to download the complete container
    there instead. mmap_mode= or a nonzero offset= forces the eager path.
  • Direct local HDF5 reads via h5py. Local .h5/.hdf5 datasets are now
    read directly through h5py without requiring kerchunk, zarr, or
    fsspec. Chunked datasets retain their native HDF5 chunk layout, while
    contiguous datasets receive automatically chosen Blosc2 cache chunks. Dataset
    attributes are loaded from HDF5 metadata, and file handles are safely closed
    when the source object is garbage-collected. Explicit refs= arguments
    continue to select the kerchunk reference reader.
  • Faster HTTP discovery and instant warm opens.
    • Cold HTTP B2Z discovery: Opening a remote B2Z archive over HTTP now
      retrieves the ZIP directory tail and remote object identity in a single
      bounded range request, halving network round-trips during initial open.
    • Warm reopens: Reopening cached B2Z, Zarr, or HDF5 sources replays persisted
      bootstrap metadata directly from the carrier file, completely bypassing
      remote discovery. Sibling HDF5 leaves from the same container under a shared
      cache_dir= reuse a single on-disk reference snapshot.
    • Whole-member prefetch for small B2Z arrays: Members up to 64 KiB are
      fetched in full on open (frame header, chunks, and trailing metadata in a
      single request). Prefetched chunks live in the standard chunk cache, count
      toward cache_bytes, respect max_cache_bytes, and can be evicted with
      trim_cache().

Readable disk caches

  • Readable source directories for cache_dir=. Persistent disk caches now
    mirror the source filename and internal dataset hierarchy (e.g.,
    hierarchy.b2z--03cc6a2f9314/d0/a1.b2nd) with a 12-character identity
    fingerprint computed from the URL and storage options. Unsafe characters are
    escaped and long components are shortened. RemoteStore caches follow the
    same naming convention. Complete containers localized with lazy=False use
    it too (e.g., temperatures.b2nd--4908c5e5602a/temperatures.b2nd).
  • Legacy hash-only cache entries are ignored rather than overwritten or deleted;
    the first open under 4.13.1 creates a new cache. Explicit cache_path= targets
    and direct opens of existing carrier files continue to work. Legacy cache
    directories can be safely deleted to reclaim disk space.

Packaging and platform support

  • Windows ARM64 builds and wheels. Added native Windows on ARM64
    (win_arm64) wheel building and testing via GitHub Actions runners,
    including build configuration adjustments for Clang and runtime fallback for
    JIT on ARM64.
  • CI test stability and doctest isolation. Enhanced test suite stability
    with logical core utilization in pytest-xdist, per-test workspace isolation
    for doctests in conftest.py, and test deadlock diagnostics.
  • Remote lazy expression examples. Added examples/remote/lazy-expr.py
    and documentation illustrating how to evaluate lazy expressions across
    remote B2Z, Zarr, and HDF5 arrays.

Bug fixes

  • Embedded frames with an offset. Opening a local file whose name looks
    like a container (.h5, .hdf5, .b2z, .zarr) with a nonzero offset
    now opens the embedded Blosc2 frame directly instead of attempting container
    parsing.
  • file:// HDF5 URLs on Windows. Dataset path separators (::) in
    file:// URLs survive URL-to-path conversion on Windows platforms
    (e.g., file:///C:/path/x.h5::/d0/a2).
  • Credential masking. RemoteArray.info and str() mask sensitive query
    parameters and credentials in signed URLs. RemoteArray.info now also works
    reliably for local B2Z sources without raising an error.
  • load_tensor() eager access. Explicitly requests eager access so it no
    longer inadvertently returns a lazy RemoteArray intermediate when loading
    known remote paths.
  • Blosc2-filtered HDF5 decoding. Fixed decoding of HDF5 datasets
    compressed with the Blosc2 filter (filter ID 32026, as written by
    hdf5plugin) when read via kerchunk. Chunks containing multi-chunk
    super-chunk frames decoded via blosc2.from_cframe() now handle the resulting
    bytes correctly, avoiding an AttributeError on .tobytes().
  • HDF5 reference snapshot publishing. Guarded reference snapshot export so
    it is only published when the leaf carrier actually holds a kerchunk map,
    fixing a crash when opening local h5py sources with a disk cache on Windows
    drive-letter paths.
  • Read-only portable carriers. Attaching a sparse runtime cache to a
    read-only legacy B2Z carrier no longer fails when refreshing bootstrap
    metadata; the session safely uses the rebuilt bootstrap in memory.