v0.8.0

ExArrow v0.8.0 Release Notes

Date: 2026-08-21

Highlights

ExArrow v0.8.0 makes Parquet a first-class citizen for larger-than-memory
workloads. Read pushdown, write compression options, footer metadata, and
multi-file directory streams let you scan only the data you need — peak memory
scales with the selected row groups, not the whole file.

• Read pushdown: :columns (projection), :row_groups (selection), and
  :filters (a predicate AST with row-group statistics pruning).
• Write options: :compression, :row_group_size, :dictionary.
• ExArrow.Parquet.Metadata: footer-only metadata without decoding rows
  (path, compression, encodings, min/max, key-value metadata).
• Multi-file streams: from_parquet_files/2, from_parquet_dir/2, plus
  Stream.close/1 when abandoning early.
• ExArrow.IPC.File.write/3 and ExArrow.RecordBatch.concat/1.

Added

• ExArrow.Stream.from_parquet/2 and Parquet.Reader.from_file/2 /
  from_binary/2 accept :columns, :row_groups, and :filters. Supported
  filter ops: :eq, :ne, :gt, :gte, :lt, :lte over integers, floats,
  UTF-8 strings, and booleans, composable via {:and, [...]} / {:or, [...]}.
  Row-group min/max statistics prune whole groups (Int32/Int64, Float32/Float64,
  Utf8, Boolean); :ne is never pruned from min/max alone. Remaining rows are
  filtered during decode via parquet-rs RowFilter. Statistics pruning resolves
  columns against Parquet leaf indices (safe for nested Struct/List schemas).
• Filter values are validated in Elixir and range-checked in the NIF before use
  (e.g. out-of-range Int32 / non-representable Float32 return {:error, _}
  instead of silent wrap/truncation).
• Parquet.Reader.read_stats/1 reports %{row_groups_total, row_groups_selected, row_groups_skipped} for a Parquet-backed stream.
• Parquet write options on Writer.to_file/4 and to_binary/3:
  :compression (:none / alias :uncompressed, :snappy, :zstd,
  {:zstd, level} with level 1..22, :lz4, :gzip), :row_group_size
  (positive integer), :dictionary (boolean).
• ExArrow.Parquet.Metadata with from_file/1 and from_binary/1 — returns
  row groups, per-column statistics (path, compression, encodings, num_values,
  min, max), and key-value metadata.
• ExArrow.Stream.from_parquet_files/2 and from_parquet_dir/2 — lazy
  per-file open in lexicographic order with schema field-name checking. Early
  Enum.take/2 does not open later files. Paths must be binaries (no silent
  to_string/1 coercion). Repeat next/1 after EOF returns nil.
• ExArrow.Stream.close/1 — stops the backing Agent for :parquet_multi
  streams (no-op for other backends). Use after partial consumption in
  long-lived processes.
• ExArrow.IPC.File.write/3 — public IPC file-format writer.
• ExArrow.RecordBatch.concat/1 — public batch concatenation.
• ExArrow.Parquet.Opts (internal) — validates read/write options in Elixir
  before reaching the NIF, returning clear {:error, message} tuples.
• livebook/05_parquet.livemd — Parquet pushdown, compressed write, multi-file
  directory, and IPC file writer, with a PyArrow side-by-side cell (Pythonx
  shares Elixir path bindings as bytes; decode before pq.read_table).
• bench/parquet_pushdown_bench.exs — pushdown filter vs full read + project
  (no public scalar-predicate kernel yet for an equivalent post-filter).
• DuckDB-generated test/fixtures/parquet_zstd.parquet and nested-struct
  pruning fixture for third-party / nested-schema regression coverage
  (thanks @mindreframer —
  #243).
• Optional arrow_testing IPC conformance suite
  (script/fetch_arrow_testing.sh + mix test --include arrow_testing) —
  local/opt-in by default; optional workflow_dispatch CI job does not gate
  PRs.
• CI: Mix.install optional-dep smoke (script/mix_install_smoke.sh), Livebook
  Hex-pin check (script/check_livebooks.sh), and livebook/00_quickstart.exs
  smoke on the test matrix.

Changed

• Native crate enabled Parquet compression codecs (snap, zstd, lz4,
  gzip / flate2-zlib-rs) and added arrow-arith so write options and
  filter evaluation work end-to-end.
• Parquet.Reader.from_file/1 / from_binary/1 are now arity-2 with an
  optional keyword list (from_file(path, opts \\ [])). Calls without options
  continue to work via the default.
• Parquet.Writer.to_file/3 / to_binary/2 are now arity-4 / arity-3 with an
  optional keyword list (to_file(path, schema, batches, opts \\ [])).
• Stream.from_parquet/1 / from_parquet_binary/1 are now arity-2 with
  optional opts; the [:ex_arrow, :parquet, :read] telemetry event now
  includes :opts, :row_groups_skipped, and :row_groups_selected.
  Multi-file batch telemetry uses the currently open path (not the full list).
• mix.exs: version bumped to 0.8.0; ExArrow.Parquet.Metadata added to
  the Parquet module group.
• README, Parquet guide, overview, CHANGELOG, and livebooks updated to
  ~> 0.8.0.

Upgrade notes

• Recommended dependency:

  {:ex_arrow, "~> 0.8"}

• The Parquet reader/writer arities changed by adding an optional opts
  keyword list at the end. Existing calls without options continue to work
  unchanged (default argument).

• ExArrow.Stream gained a :parquet_multi backend for multi-file streams.
  The resource field may now hold an Agent pid for that backend; code that
  pattern-matched on resource being a plain reference should use the public
  Stream API instead. Call Stream.close/1 when abandoning a multi-file
  stream early from a long-lived process.

• :uncompressed is accepted as an alias for :none on write compression.
  Zstd levels must be in 1..22.

Verification performed

• mix compile --warnings-as-errors, mix format --check-formatted,
  mix credo --strict, mix sobelow --exit --skip, script/check_version,
  script/check_livebooks.sh, mix dialyzer, mix test,
  mix docs --warnings-as-errors, and CI matrix (Elixir 1.18–1.20 / OTP 27–29,
  Mix.install smoke, ADBC integration) — green on merge of
  #244.

• README updated by @thanos in #262

Full Changelog: v0.7.2...v0.8.0