Skip to content

v0.10.0

Choose a tag to compare

@github-actions github-actions released this 26 Jun 18:33
Immutable release. Only release title and notes can be modified.

A vortex.zstd overhaul: compression now runs through FFM bindings to the native libzstd, gaining framed (sliceable) payloads, nullable-column support, and shared-dictionary decode. Alongside it, zone-map pruning is fixed to compare in the column's type domain.

Added

  • DType.isUnsigned()true for the unsigned integer primitives (U8U64), false otherwise. (#159)
  • The vortex.zstd encoder now writes nullable columns (primitive and utf8/binary): null positions are stripped before compression and validity is emitted as a Bool child, matching the Rust reference layout. When vortex.zstd is the configured encoder, nullable primitive columns route to it directly instead of being wrapped in vortex.masked.
  • new ZstdEncodingEncoder(valuesPerFrame) splits the payload into independently compressed frames of valuesPerFrame values each (one ZstdFrameMetadata per frame), letting a slice scan decompress only the frames overlapping its row range. The no-arg constructor still emits a single frame. (#170)

Changed

  • The vortex.zstd encoding now compresses and decompresses through io.github.dfa1.zstd:zstd (FFM bindings to the native libzstd) instead of io.airlift:aircompressor-v3. Consumers of vortex.zstd declare a single dependency, io.github.dfa1.zstd:zstd-platform, which transitively brings the zstd binding plus the native libzstd for every supported platform (replacing the former per-platform zstd-native-<platform> artifacts).

Fixed

  • vortex.zstd segments compressed with a shared (trained) dictionary now decode, via the native libzstd dictionary support, instead of being rejected. The upstream zstd.vortex compatibility fixture is read end-to-end and matches the Rust reference. (#104)
  • Writing a nullable Utf8/Binary column no longer throws NullPointerException (or silently drops nulls): nullable string columns now carry their validity like nullable primitives and round-trip through vortex.masked. As a result they decode as MaskedArray (validity + values child) rather than a bare VarBinArray. (#168)
  • CSV export now handles nullable columns (MaskedArray): null rows export as an empty field instead of failing with "unsupported array type for CSV export". (#168)
  • Zone-map pruning now compares filter values in the column's type domain rather than by the boxed value's type. A predicate whose value is boxed at a different width (e.g. Integer on an I64 column) — or any value on a U64 column — previously pruned nothing and silently degraded to a full scan; it now prunes correctly (unsigned columns by unsigned order). As part of this, a filter value genuinely incomparable to its column (e.g. a String against a numeric column) now raises VortexException during the scan instead of silently disabling pruning — a behaviour change for callers that relied on the previous silent full scan. (#159)