Skip to content

Releases: d-v-b/refkv

Release list

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 31 Oct 15:29
10afa20

0.2.0 (2026-10-31)

⚠ BREAKING CHANGES

  • zarr: archives name numcodecs.zlib for zlib streams and refkv.tiff_lzw for QPTIFF LZW tiles; refkv no longer registers zlib, imagecodecs_lzw, imagecodecs_jpeg, imagecodecs_jpeg2k or imagecodecs_jpegxr (the last three come from imagecodecs-zarr, in the images extra).
  • say zarrify for virtualize, in the code, the CLI, the spec and the docs (#112)
  • mp4: cut video chunks at keyframes and decode a chunk's planes once, as a stream (#107)
  • ir: add the refkv.stream codec, port MP3 onto it, and virtualize MP4 files as the MP4 profile (§18) (#99)
  • reader: read http(s) archives with refkv's own HTTP client, and make obstore an extra (#72)
  • python: make python/ a self-contained package versioned from git tags (#71)

Features

  • cli: add vzip convert, a Rust port of the converter that writes identical archives (#78) (a6fb670)
  • demo: add a QuickTime .mov from DANDI 001538 to the MP4 examples (#154) (3fabeb2)
  • demo: add examples of LIF, QPTIFF, HDF5, NumPy, MP3 and FLAC files (#160) (099d5b2)
  • demo: add NOAA HRRR and GFS forecasts as GRIB examples (#170) (4b3d193)
  • demo: link a hosted .vzip archive as a plain vzip: Neuroglancer source (#126) (1f5ffbc)
  • demo: make a DANDI behavior video the MP4 example (#152) (2fa1ea1)
  • demo: show a Python script that builds the same Zarr hierarchy (#169) (cbe7ad0)
  • demo: show an MP4 file's video in Neuroglancer, with a SLEAP example (#108) (463c316)
  • dicom: zarrify JPEG Lossless frames as imagecodecs_ljpeg chunks (#163) (97c4ef0)
  • flac: zarrify FLAC files as virtual shards of frames, with a shared shard pass (#121) (df14a47)
  • grib: zarrify GRIB2 files, each message a chunk decoded by refkv.grib (#166) (0dd9063)
  • hdf5: zarrify HDF5 files, NWB first among them, in hdmf-zarr's Zarr v3 form (#143) (8a56953)
  • ir: add the refkv.stream codec, port MP3 onto it, and virtualize MP4 files as the MP4 profile (§18) (#99) (8f79104)
  • ir: virtualize MP3 files through the Rust IR, as the MP3 profile (§16) (#91) (0b127f3)
  • ir: virtualize MRC2014 files through the Rust IR, as the MRC profile (revision 28) (#83) (5b43073)
  • ir: virtualize Olympus VSI slides through the Rust IR, as the VSI profile (revision 29) (#87) (8ab760f)
  • js: build refkv_embed.js, the virtualizer and FITS tile decoder for a host that reads the input itself (#109) (62af0f1)
  • lif: zarrify Leica LIF files, as the LIF profile (#118) (4a14bec)
  • mp4: cut video chunks at keyframes and decode a chunk's planes once, as a stream (#107) (fd78ebe)
  • mp4: zarrify H.264 tracks whose pictures are presented before their sync sample (#110) (8fec438)
  • mp4: zarrify HEVC video tracks, 8- and 10-bit 4:2:0 (#151) (99eda76)
  • ndpi: zarrify JPEG XR levels, with the pixels their streams hold (#125) (3e3cdc1)
  • npy: present the deflated members of .npz archives, as np.savez_compressed writes them (#158) (9d409d9)
  • npy: zarrify NumPy .npy files and the stored arrays of .npz archives (#156) (2780e2e)
  • nrrd: zarrify NRRD files, attached or detached, raw or gzip, as the NRRD profile (#119) (465d575)
  • phkv: add the perfect-hash binding (Part 3, draft), standalone and hybrid with vzip (#79) (6e7776e)
  • python: build free-threaded wheels for CPython 3.14t and 3.15t (#174) (1f5b7d1)
  • reader: read http(s) archives with refkv's own HTTP client, and make obstore an extra (#72) (a165682)
  • tiff: zarrify images in strips and LZW, as the QPTIFF profile does (#137) (3b4cbd8)
  • tiff: zarrify images with a horizontal predictor through imagecodecs_delta (#139) (242ae27)
  • types: type-check the repository with pyrefly's strict preset (#116) (eefa6bd)
  • virtualize: add an omero block to TIFF and NDPI images, with channel labels and colors (#95) (7d051e3)
  • virtualize: add the QPTIFF profile, with LZW tiles and strips (#84) (850969d)
  • virtualize: sample the omero windows of TIFF and QPTIFF images from the coarsest level (#98) (01c4200)
  • vsi: zarrify lossless JPEG tiles, and ETS headers that list no dimension sizes (#162) (52304c5)
  • zarr: expose refkv's codecs to zarr through entry points, and rename the clashing ones (#136) (1bdab30)
  • zarrify: add zarrify_async and zarrify_store_async, which keep the event loop free (#175) (ca01733)
  • zarrify: name an input's profile with a source or store class, and choose its layouts (#114) (db24287)
  • zarrify: pad NIfTI, MRC and FITS edge chunks with one shared zero data source (#120) (b5f0eda)

Bug fixes

Read more

v0.1.0

Choose a tag to compare

@d-v-b d-v-b released this 13 Oct 20:03
07ab50c

What's Changed

  • feat(spec): define vzip format version 0 (provisional) and converge it through independent implementations by @d-v-b in #1
  • feat(web): serve virtualized TIFFs as plain Zarr from the browser by @d-v-b in #2
  • chore: license the repository under MIT OR Apache-2.0 by @d-v-b in #6
  • docs: credit kerchunk and VirtualiZarr as the basis of vzip by @d-v-b in #7
  • docs: link the live demos from the README by @d-v-b in #8
  • fix(web): wait for the vzip worker itself before using the demo by @d-v-b in #9
  • feat: virtualize Nikon ND2 files and coalesce nearby range reads by @d-v-b in #10
  • feat(virtualize): specify virtualization of TIFF, SVS, NDPI and ND2 in VIRTUALIZE.md by @d-v-b in #17
  • feat(convert): write Zarr consolidated metadata into the root zarr.json by @d-v-b in #23
  • feat(comparison): measure kerchunk JSON, kerchunk Parquet, Icechunk and vzip on one virtual dataset by @d-v-b in #21
  • perf(store): reuse HTTP connections for url sources by @d-v-b in #24
  • feat(virtualize): virtualize DICOM, NIfTI and Imaris IMS files by @d-v-b in #20
  • feat(virtualize): virtualize N5 and Zarr v2 stores as Zarr v3 by @d-v-b in #30
  • feat(virtualize): migrate OME-Zarr 0.4 stores to OME-Zarr 0.5 in place by @d-v-b in #31
  • perf(virtualize): hold shared JPEG headers in data sources by @d-v-b in #27
  • test(virtualize): shrink the store fixtures, and fix two store review findings by @d-v-b in #29
  • feat(virtualize)!: specify each format's Zarr layout and source metadata as a Zarr convention by @d-v-b in #33
  • docs(virtualize): add a user-story README per format by @d-v-b in #35
  • fix(web): open the demo inside a translated image's data by @d-v-b in #36
  • perf(virtualize): read store metadata documents concurrently by @d-v-b in #37
  • feat(virtualize)!: virtualize TIFF, ND2 and CZI through a Rust IR core, add CZI and SAFE, and default readers to a secure policy by @d-v-b in #40
  • refactor!: reorganize the repository by component and language, and merge each format's profile and convention by @d-v-b in #43
  • feat(demo): add examples for the new formats, and show Sentinel-2 products on a map by @d-v-b in #44
  • feat(demo): group the examples by format, with a badge and a link to each format's spec by @d-v-b in #45
  • fix(virtualize)!: take an IMS channel's color from its lookup table by @d-v-b in #46
  • feat(demo): show every image of 2 to 16 channels as one layer, however its channels are chunked by @d-v-b in #48
  • docs: link every section reference to its heading by @d-v-b in #50
  • fix(virtualize): read store inputs and every file input's request under the reader policy by @d-v-b in #41
  • fix(test): serve CONNECT tunnels in the store policy test's proxy, and test Node 24 and 26 by @d-v-b in #52
  • docs(spec): split the archive spec into the reference value type and its ZIP binding by @d-v-b in #51
  • feat(python)!: add ReferenceStore, an in-memory store of reference values saved as a vzip archive, with a virtual-datasets docs page by @d-v-b in #53
  • build!: make zarr an optional dependency, behind a vzip.zarr adapter and the vzip[zarr] extra by @d-v-b in #55
  • docs(spec): explain where a reference's bytes that no source holds go by @d-v-b in #54
  • feat(spec)!: write the ZIP64 end records in every archive and allow large entries (revision 11) by @d-v-b in #5
  • feat(ir)!: describe a store's objects as the sources of one IR by @d-v-b in #56
  • feat(virtualize)!: virtualize DICOM through the Rust IR, and make its mirror DICOM's source metadata (revision 26) by @d-v-b in #57
  • feat(ir)!: virtualize NDPI through the Rust IR, with its IR mirror as source metadata (revision 26) by @d-v-b in #58
  • build!: ship the Rust core inside vzip's wheel, and publish wheels to PyPI on a version tag by @d-v-b in #63
  • feat(virtualize)!: move NIfTI onto the Rust IR, and make its mirror the source metadata (revision 26) by @d-v-b in #59
  • feat(ir)!: virtualize Sentinel-2 SAFE products through the Rust IR, in both forms by @d-v-b in #60
  • feat(ir)!: virtualize Imaris IMS through the Rust IR, with the file's mirror as vzip_source (revision 26) by @d-v-b in #62
  • feat(ir)!: move the Zarr v2 store profile onto the multi-source Rust IR (VIRTUALIZE revision 26) by @d-v-b in #61
  • feat(python)!: return a ReferenceStore from vzip.virtualize.virtualize by @d-v-b in #64
  • test: collect python/tests in one pytest run by @d-v-b in #68
  • feat(ir)!: virtualize N5 stores through the multi-source Rust IR, with the store's mirror as vzip_source (revision 26) by @d-v-b in #65
  • feat(ir)!: move the OME-Zarr store profile onto the Rust IR as a layer on Zarr v2 (VIRTUALIZE revision 26) by @d-v-b in #66
  • refactor(virtualize): reject an unrecognized file from the IR sniff, dropping _virtualize_file by @d-v-b in #67
  • refactor!: rename vzip to refkv, move the ZIP binding under refkv.vzip, and split its spec (revision 27) by @d-v-b in #69

Full Changelog: https://github.com/d-v-b/refkv/commits/v0.1.0