0.91.0

Changes

⚠️ Breaks

• Add shared conversion to canonical decimal arrays or Widen (#11005) @connortsui20
• Reject nullable nested levels in list-based spatial types (#11183) @connortsui20
• Preserve extension literals in DataFusion comparisons (#11067) @connortsui20
• Require matching dtypes for extension comparisons (#11016) @connortsui20
• Change extension dtype nullability without validating again (#11012) @connortsui20
• Separate the read and write IDs of array plugins (#11057) @connortsui20
• Vortex spark integration is a fileformat not a tableprovider (#9658) @robert3005
• Return errors for invalid temporal types and values (#11015) @connortsui20
• DictStrategy probes a fixed row sample instead of the first chunk (#10927) @mhk197
• Layout plugins read and write more than one wire ID (#10704) @mhk197
• Make uncompressed_size_in_bytes always exact and drop its options (#10938) @joseph-isaacs
• Add a canonical decimal width and import decimals at it (#10897) @connortsui20
• Move RunEnd into vortex-array (#10703) @mhk197
• Make protobuf metadata fields crate-private (#10831) @connortsui20
• Decode stored DecimalType and PType values strictly (#10755) @connortsui20
• Return a borrowed scalar from as_constant (#10790) @robert3005
• Simplify error types, remove kind from struct (#10080) @robert3005
• perf(cuda): migrate consumers to vortex-cuda-driver (#10722) @0ax1

✨ Features

• feat(array): migrate simple encodings to scalar probes (#10859) @jiengup
• Explorer: draw it with the Vortex design system, and build it with Bun (#11193) @lwwmanning
• Property-based tests for Mask (#11167) @myrrc
• Replace layout lowering with a VTable::new_plan method (#11200) @joseph-isaacs
• Expose the existing footer segment end offset (#11196) @brancz
• test: remove alp tests which are covered by other tests (#11155) @myrrc
• test: remove vortex-duckdb tests which are covered by other tests (#11153) @myrrc
• Support decimal values in Widen (#10946) @connortsui20
• Add ScalarFnVTable::execute_step for iterative scalar fn execution (#11100) @joseph-isaacs
• Read Vortex file column statistics through vortex-jni (#10863) @robert3005
• Select editions from JNI and Python, and write files for readers before vortex.zoned (#10805) @robert3005
• fix(python): read pyarrow files positionally in open_readable (#11074) @jackylee-ch
• feat(spark): push nested struct pruning down to the native scan (#10957) @ChristopherWirt
• Add serialization for aggregate functions (#11048) @gatesn
• Record the uncompressed size in file stats when editions allow it (#10923) @joseph-isaacs
• Add uncompressed_size_in_bytes to the preview edition (#10922) @joseph-isaacs
• Execute interleaved decimal arrays (#10940) @connortsui20
• Add a scan for the range of valid decimal values (#10939) @connortsui20
• Add an in-memory Widen encoding for integers (#10864) @connortsui20
• Add a naive Unary encoding in a new vortex-elias-fano crate (#10609) @rapour
• duckdb: support windows-x86 and windows-arm64 (#9570) @myrrc
• string-bench: benchmark onpair-16 and add the OnPair paper datasets (#10920) @joseph-isaacs
• Add for_each_unset_index to BitBuffer (#10866) @rapour
• Push DataFusion dynamic filters through Scan API expression trees (#10895) @gatesn
• Skip file filters proven true until their dynamic producer changes (#10894) @gatesn
• Derive live statistics proofs for dynamic expressions (#10869) @gatesn
• Add generic dynamic expressions with generation tracking (#10865) @gatesn
• feat(fsst): Compute uncompressed size without decoding (#9755) @lwwmanning
• Add list_transform (#10503) @mhk197
• Rebuild lists over only the elements valid lists reference (#10743) @mhk197
• Expose aggregate statistics via Scan API (#10838) @gatesn
• Add an unstable edition that includes all unreleased and upcoming encodings (#10558) @robert3005
• Drive V2 scans in the DuckDB extension with a custom driver (#10809) @joseph-isaacs
• Pass VortexSession to plan scan and optimizer (#10794) @myrrc
• duckdb: export DECIMAL and TIME_NS min/max statistics (#10635) @jackylee-ch
• Statistics for nested columns (#9775) @thorfour
• feat(cuda): track device memory accesses across streams (#10721) @0ax1
• feat(cuda): add driver context and resource ownership (#10720) @0ax1
• feat(cuda): add minimal CUDA driver bindings (#10719) @0ax1
• Add Eval(Eval()) and Eval(pack()) reduce rules (#10732) @myrrc
• Add lambdas and variables for higher-order scalar functions (#10502) @mhk197

🚀 Performance

• Keep same-scale decimal widening casts lazy (#11006) @connortsui20
• perf(spark): split large files into row ranges (#11118) @gitRasheed
• perf(fastlanes): push FoR ordering comparisons to the encoded values (#10147) @robert3005
• Count each run's selected rows with one pass over the RunEnd filter mask (#11221) @joseph-isaacs
• Drop null patch elements when sparse lists share their elements (#11182) @mhk197
• Faster Spark reads and smaller JNI and stats fixes (#11109) @robert3005
• Optimize arrays top down as execution reaches them, and retry a parent when its children change (#11090) @robert3005
• refactor(cuda): avoid access tracking for address-only lookups (#10884) @0ax1
• Store plan children in fixed-position slots (#11178) @joseph-isaacs
• Push decimal aggregates through Widen (#11004) @connortsui20
• Reuse compact elements when canonicalizing sparse lists (#11173) @gatesn
• perf(spark): append rows straight into the Arrow batch (#11152) @gitRasheed
• Export decimal Widen arrays to Arrow with one copy (#10994) @connortsui20
• Push down list element casts and bool/int/uint casts to Float64 in DataFusion (#11161) @mhk197
• Push decimal comparisons, between, and fills through Widen (#11003) @connortsui20
• perf(decimal-byte-parts): compare wide decimals through their parts (#10146) @robert3005
• perf(alp): push nullable and patched comparisons to the encoded ints (#10141) @robert3005
• perf(sequence): answer ordering and inequality comparisons directly (#10144) @robert3005
• Record whether an array has been optimized (#11089) @robert3005
• Default duckdb-bench to the V2 scan for TPC-DS and ClickBench (#11047) @joseph-isaacs
• Speed up the decimal widening copy (#10989) @connortsui20
• Speed up V2 scans and default datafusion-bench to V2 for TPC-DS and ClickBench (#10799) @joseph-isaacs
• Separate RowFn batch inspection from execution planning (#10899) @connortsui20
• Push slices through filtered arrays (#10959) @joseph-isaacs
• Avoid a per-run zip_eq call in run-end bool decoding (#10962) @connortsui20
• Execute Delta iteratively (#10208) @joseph-isaacs
• perf(bytebool): take through the fallible lane kernels (#10143) @robert3005
• perf: transfer statistics to executed arrays without cloning the set (#10241) @joseph-isaacs
• Run Widen arithmetic, cast, fill null, zip and between on the values (#10891) @connortsui20
• Compute Widen aggregates on the values (#10890) @connortsui20
• duckdb: serialize table function state (#10929) @myrrc
• For decompression works on owned arrays (#10256) @robert3005
• Reduce SequenceId universe lock contention (#10867) @robert3005
• fix: Convert V2 scan batches to Arrow on the driver's tasks (#10861) @joseph-isaacs
• Use NEON packing for wide primitive array comparisons (#10843) @connortsui20
• Dispatch structural kernels by physical width (#10841) @connortsui20
• perf: skip parent kernel lookups for encodings that register none (#10239) @joseph-isaacs
• Compose and drive a V2 scan in DataFusion (#10708) @joseph-isaacs
• Widen dictionary codes in Dict::fill_null (#10806) @ump45nose
• Attach RowFn validity directly to canonical output (#10742) @connortsui20
• fix: bit-packed arrays cast match more often (#10803) @joseph-isaacs
• Plan a V2 scan as a chain of planners from one prepared scan (#10707) @joseph-isaacs
• Pack float32 comparison results into bitmap words (#10741) @connortsui20
• perf(cuda): release closed mappings off the decode thread (#10718) @0ax1
• perf(cuda): reuse pinned buffer completion events (#10713) @0ax1
• perf(cuda): reuse file mappings (#10717) @0ax1
• perf(cuda): copy cached file pages directly on the GPU (#10716) @0ax1
• perf(cuda): disable timing on pinned transfer events (#10714) @0ax1
• perf(cuda): read pooled files in 8 MiB chunks (#10715) @0ax1
• Compare timestamp parts without evaluating each comparison twice (#10684) @robert3005
• Optimise take on dictionaries with constant codes (#10643) @robert3005
• Add Filter/Split/Project planner and add Datafusion usage of this (#10706) @joseph-isaacs

🐛 Bug Fixes

62 changes

• Peel Arrow dictionaries before Arrow importer plugins run (#11185) @connortsui20
• Display dates beyond jiff's range without panicking (#11187) @connortsui20
• Return an error for decimal scalars outside the precision (#10767) @ump45nose
• Require strictly increasing patch indices (#11181) @mhk197
• fix(array): accept timestamps beyond jiff's range (#10953) @gitRasheed
• fix: preserve sliced patch offsets during deserialization (#11168) @blaginin
• Fix dictionary code width for nullable primitive values (#11165) @blaginin
• Cast bool scalars to primitive types (#11171) @mhk197
• Avoid rebuilding constant null and empty lists for Arrow export (#11172) @gatesn
• Reduce constant projections through select and extension storage (#11056) @gatesn
• fix(layout): rebase sliced list chunks before writing the list layout (#11082) @gitRasheed
• fix(cuda): report device errors to stream callback waiters (#10876) @0ax1
• fix(cuda): reject an empty stream pool when it is created (#10875) @0ax1
• fix(cuda): make loading a module image unsafe (#10874) @0ax1
• fix(cuda): report missing driver library as an error (#10873) @0ax1
• fix(cuda): do not leak contexts when construction fails (#10872) @0ax1
• fix(cuda): harden driver error handling, access tracking and host copies (#10871) @0ax1
• fix(jni): retain Java Arrow buffers independently (#11069) @ChristopherWirt
• duckdb: report distinct count based on min/max (#11130) @myrrc
• fix(duckdb): handle table filters in aggregate pushdown (#11071) @jackylee-ch
• fix(duckdb): escape LIKE metacharacters in pushed string filters (#11072) @jackylee-ch
• fix(jni): attach native worker threads to the JVM as daemons (#11085) @gitRasheed
• Validate extension storage and return scalar read errors (#11014) @connortsui20
• fix(python): keep the scheme of object store paths in the Ray datasource (#11046) @jackylee-ch
• fix(spark): resolve pruned nested fields against each file's struct (#11083) @gitRasheed
• fix(spark): let abort close every writer after a failed write (#11084) @gitRasheed
• fix(jni): release the schema export in VortexWriter.build() (#11081) @gitRasheed
• fix(python): report the scan's row count and columns from the Ray datasource (#11070) @jackylee-ch
• fix(spark): read every column as nullable (#11020) @ChristopherWirt
• Clarify DCO sign-off requirements and agent attribution (#11001) @joseph-isaacs
• fix(duckdb): don't drop or reject filters on function arguments Vortex can't convert (#10958) @jackylee-ch
• Update libz-sys to 1.1.31 to fix RUSTSEC-2026-0418 (#10963) @robert3005
• fix(alp): bounds-check ALP-RD stored metadata and left-parts codes (#10760) @devtechedge
• fix: add explore's CSP and fix docs search and previews (#10951) @lwwmanning
• Build precision 19 decimal scalars at i128 (#10903) @connortsui20
• Fix exact/inexact mixing in uncompressed_size_in_bytes (#10925) @joseph-isaacs
• fix: keep nullability when short-circuiting AND/OR against a constant (#10896) @robert3005
• feat(python): fail soft on unsupported Polars predicates in to_polars (#10414) @robert3005
• perf: collapse single-value partitions instead of a direct struct projection path (#10536) @robert3005
• Expose DataFusion output partitions and proven ordering (#10901) @gatesn
• Honour NaN options in min/max over float-backed extension types (#10851) @robert3005
• fix: preserve strict sortedness for singleton null batches (#10889) @connortsui20
• fix: keep empty encoded batches neutral in IsConstant (#10893) @connortsui20
• Speed up file writes: parallel file stats, single min/max pass (#10856) @robert3005
• perf(layout): prune multi-field boolean predicates in the struct reader (#10689) @MichaHoffmann
• Compute legacy file stats from the aggregate stats accumulators (#10846) @robert3005
• Preserve ordered Scan API partitions in the v2 DataFusion adapter (#10062) @vortex-claude[bot]
• DataFusion v2 expression converter injection (#10836) @gatesn
• Fix datafusion v2 statistics precision (#10835) @gatesn
• Reject raw serialization of sliced PCO arrays (#10834) @danking
• fix(array): between over a constant null array is null, not false (#10744) @gitRasheed
• fix(spark): support Spark 4.2 in the current connector (#10792) @ChristopherWirt
• Avoid truncating dictionary fill codes (#10745) @connortsui20
• fix: preserve codes nullability when slicing dict with a constant code (#10772) @robert3005
• FoRScheme: declare and refine its bit-packed child (#10757) @mhk197
• fix(cuda): retain pinned sources across transfer failures (#10712) @0ax1
• Dispatch count_ones using fearless_simd (#10683) @robert3005
• fix(cuda): bound concurrent stream-pool initialization (#10711) @0ax1
• fix: update ctutils for RUSTSEC-2026-0385 (#10726) @0ax1
• fix: write empty nested list streams with structural layouts (#10533) @danking
• Propagate child writer errors from the struct writer (#10710) @robert3005
• fix(spark): return the requested field when Spark prunes nested fields (#10724) @gitRasheed

📖 Documentation

8 changes

• docs(arrow): explain why dictionary imports need special handling (#11233) @connortsui20
• docs(cuda): clarify mapped-page eviction (#10882) @0ax1
• docs llms.txt links to the benchmark and homepage llms.txt files (#11177) @lwwmanning
• docs: add robots.txt, a sitemap, llms.txt, and a favicon (#11166) @lwwmanning
• Document temporal storage and binary comparison execution (#11099) @connortsui20
• Better READMEs and user docs (#10960) @myrrc
• Document the nullability contract of cast rules (#10798) @connortsui20
• Describe scheme variants in the versioning docs (#10747) @connortsui20

🧰 Maintenance

78 changes

• Avoid allocating Vec of chunks just to copy it in ChunkedArray constructor (#9922) @robert3005
• Execute filter scalar fns' inputs iteratively with execute_step (#11154) @joseph-isaacs
• test(cuda): encode ALP test inputs with the test's own context (#10881) @0ax1
• refactor(cuda): slice mutable views through one implementation (#10880) @0ax1
• Benchmark sparse-list canonicalization with a null fill (#11175) @gatesn
• Duplicate handling in Mask::from_excluded_indices (#11164) @blaginin
• Dispatch zip kernels on either value branch (#11138) @blaginin
• bench: add sparse-wide datasets to compression suite (#11150) @gatesn
• refactor(cuda): keep inflight pinned buffers unwrapped (#10878) @0ax1
• Lock file maintenance (#11080) @renovate[bot]
• Lock file maintenance (#11079) @renovate[bot]
• Lock file maintenance (#11078) @renovate[bot]
• Run the remaining benchmarks on the V2 scan (#11061) @joseph-isaacs
• Add execution path consistency assertions (#11013) @connortsui20
• Run the FineWeb benchmark on the V2 scan (#11055) @joseph-isaacs
• Fix benchmark runner labels exceeding 255 character limit (#11054) @joseph-isaacs
• Optimize VCF parsing with concurrent range requests and parallel row group processing (#10984) @joseph-isaacs
• Ingest string-bench results into the compression tables (#11044) @joseph-isaacs
• Run CUDA Rust and Python tests in parallel steps (#11039) @joseph-isaacs
• chore: let Renovate propose minor and patch updates to GitHub Actions (#11041) @lwwmanning
• chore: let Renovate propose grouped major updates for vortex-web (#11042) @lwwmanning
• Replace Kanal with async-channel (#11040) @gatesn
• fix(bench): retry the post-ingest database write across RDS restarts (#11018) @lwwmanning
• Fix Arrow export of constant null variants (#10973) @blaginin
• Lock file maintenance (#11011) @renovate[bot]
• Write the list layout only for statpopgen and polarsignals benchmark data (#11035) @mhk197
• Lock file maintenance (#11010) @renovate[bot]
• Lock file maintenance (#11009) @renovate[bot]
• Enable the list layout in benchmarks through a bench-only edition (#11030) @mhk197
• ci: cargo hack runs only check and not build (#10845) @robert3005
• chore: wait 7 days for new releases and fix Renovate's lock file PRs (#10997) @lwwmanning
• chore: refresh lockfiles to clear Dependabot alerts (#10999) @lwwmanning
• Benchmark the copy of decimal values into the canonical width (#10990) @connortsui20
• Resolve three small TODO(connor) items (#10974) @connortsui20
• Add a toggle for writing nested file statistics (#10972) @robert3005
• Extend the RunEnd filter benchmark (#10978) @robert3005
• Update dependency filelock to v4 [SECURITY] (#10955) @renovate[bot]
• vortex-bench: run ClickBench on its first N files (#10936) @mprammer
• bench: name the data directory when capturing file sizes (#10935) @mprammer
• chore: remove the Cloudflare Pages deploys and update bench paths (#10913) @lwwmanning
• Remove repeated elements from IN lists (#10685) @robert3005
• Add compute benchmarks for Widen (#10888) @connortsui20
• Allocate more kernel outputs through the session allocator (#10622) @robert3005
• string-bench: run the new inputs in CI (#10924) @joseph-isaacs
• Enable preview and unstable editions in the fuzzer (#10840) @robert3005
• Upgrade fsst-rs to 0.7.0 and onpair to 0.3.0 (#10909) @robert3005
• Update dependency com.google.guava:guava to v33.7.2-android [SECURITY] (#10870) @renovate[bot]
• Lock file maintenance (#10788) @renovate[bot]
• Skip codegen of unreachable type dispatch arms (#10740) @robert3005
• Drop direct arrow-select dependency (#10826) @robert3005
• Remove unused dictionary layout metadata (#10823) @connortsui20
• Pin cc to 1.8.0 to fix the Windows CI build (#10815) @robert3005
• Widen list take piece bounds to u64 before dispatch (#10738) @robert3005
• Share FastLanes compare kernels between operators and signedness (#10739) @robert3005
• Build arrays through non-generic constructors (#10737) @robert3005
• Dispatch filter take of fixed-width values on byte width (#10736) @robert3005
• Skip codegen of unreachable int to decimal carrier arms (#10735) @robert3005
• Update Cargo.lock to pick up security fixes (#10810) @robert3005
• Add decimal benchmarks for DecimalByteParts, casts and exports (#10752) @connortsui20
• Add compat fixtures for old decimal forms (#10753) @connortsui20
• ci: disable CodSpeed process isolation for Rust simulation benchmarks (#10796) @robert3005
• Consolidate python script, formatting and c++ lint tests into one job (#10729) @robert3005
• chore(deps): update rui314/setup-mold digest to 339e96b (#10778) @renovate[bot]
• chore(deps): update swatinem/rust-cache digest to d1527c8 (#10779) @renovate[bot]
• chore(deps): update pypa/gh-action-pypi-publish digest to 7245906 (#10777) @renovate[bot]
• chore(deps): update taiki-e/install-action digest to 1e29709 (#10780) @renovate[bot]
• chore(deps): update dtolnay/rust-toolchain digest to b725eda (#10776) @renovate[bot]
• chore(deps): update rust crate criterion to v6 (#10784) @renovate[bot]
• chore(deps): update codspeedhq/action digest to 78102a4 (#10775) @renovate[bot]
• chore(deps): update actions/setup-python digest to 6d4fd22 (#10774) @renovate[bot]
• chore(deps): update actions/setup-java digest to b843ebf (#10773) @renovate[bot]
• chore(deps): update rust crate divan to v6 (#10785) @renovate[bot]
• Report one-row Sequence arrays as sorted for any multiplier (#10602) @ump45nose
• Move expression analysis, rewrites and statistics pruning into vortex-expr (#10734) @robert3005
• Remove array_session (#10681) @robert3005
• ci: cache Python wheel compilation on prebuilt runners (#10727) @robert3005
• ci: use the SQL fixture profile for Vortex conversion (#10728) @robert3005
• Parallelize TPC-DS generation within each table (#10692) @robert3005