Skip to content

v0.7.2

Choose a tag to compare

View all tags
@dfa1 dfa1 released this 16 Jun 19:56
v0.7.2
cf80a78

CLI usability + reader robustness on real-world files (NYC Yellow Taxi).

Added

  • CLI view <file> — scrollable Excel-like grid TUI. Streams rows on demand via a new LazyGridSource (one live chunk at a time, formats only the visible window). Title bar shows chunk K/N. Quit with q / Esc.
  • CLI export writes to a derived <name>.csv next to the input file by default, with a stderr progress bar mirroring the import flow. Use export <file.vortex> - to stream to stdout, or export <file.vortex> out.csv for an explicit path.
  • Reader ScanIterator.chunkRowCounts() — returns per-chunk row counts by walking the layout tree, no value decode. Used by the view TUI to plan navigation.
  • Lazy vortex.decimal decode via new LazyDecimalArray record — zero-copy mmap slice + per-row BigDecimal materialisation. Replaces the GenericArray wrapper.
  • 7 Offset*Array records (Long / Int / Short / Byte / Double / Float / Bool) + VarBinArray.SlicedMode for offset-based slicing of pre-decoded shared arrays.

Fixed

  • Per-column chunking alignment. Files where one column has 1 mega-flat and another has N small flats (e.g. NYC Yellow Taxi 2024-01: 2.96M-row VendorID next to 23 × 131072-row datetime flats) now decode the wide column once into a sharedArena and slice it per chunk via Offset*Array. Previously the scan iterator emitted a single chunk whose datetime columns were the first 131072 rows only — silently dropping 95.6 % of the file.
  • FrameOfReferenceEncodingDecoder now uses the arena variant of ArraySegments.of, so lazy children (e.g. LazyRunEndLongArray) materialise instead of throwing "no primary segment".

Docs

  • Compatibility table refreshed: constant, varbinview, alprd, datetimeparts, decimal_byte_parts, decimal now reflect their shipped Lazy shape; container encodings (list / listview / fixed_size_list) marked Lazy (inherit child shape); patched pinned Materialized with reasoning.
  • New ADR 0013 — policy for dropping Materialized fallbacks once Lazy ships.

Maven Central: io.github.dfa1.vortex:vortex-reader:0.7.2 (and vortex-writer, vortex-cli, etc.).

Assets 2
Loading