0.89.0

Changes

⚠️ Breaks

• C++ FFI field access method doesn't require a session (#10611) @robert3005
• Pass an execution context to array value display (#10583) @robert3005
• Take a session in vx_array_get_field and vx_array_invalid_count (#10589) @robert3005
• duplicate ZstdBuffer's validity into a slot instead of using the legacy session (#10586) @robert3005
• Take an execution context in CanonicalCudaExt::into_host (#10590) @robert3005
• Blocked FoR: store a global reference scalar as metadata rather than child (#10222) @mhk197
• Move Shared array out of vortex-array into vortex-layout (#10495) @robert3005

✨ Features

• Convert FixedSizeList scalars to and from DuckDB ARRAY values (#10629) @robert3005
• feat(aggregate): add ExtensionStoragePushdown to push an aggregate into an extension type's storage (#10566) @joseph-isaacs
• Expose list membership with SQL null semantics in Java and Python (#10053) @robert3005
• Duckdb 2.0 (#9141) @myrrc
• feat(array): enable CaseWhen expression serialization (#10604) @blaginin
• feat(sparse): push LIKE through Sparse arrays (#10538) @robert3005
• feat(layout): zoned pruning, shares, and dictionary value reuse in physical plans (#10462) @joseph-isaacs
• perf(python): release the GIL once per scan export (#10497) @robert3005
• feat(python): open local files as memory maps (#10498) @robert3005
• Optimize list_contains with prepared constant sets (#10052) @robert3005
• feat: expose Spark scan schema and pushed predicates (#9956) @xiaoh1024
• Blocked BitPacked: Add Compressor support via BitPackingScheme mode (#10245) @mhk197
• Support Polars 2 (#10398) @robert3005
• feat(layout): compile evals, row indices, and takes to pipelines (#10393) @joseph-isaacs
• feat(layout): compile segment scans, filters, concatenations, and packs to pipelines (#10392) @joseph-isaacs
• feat(layout): add push-based pipelines and the scan that runs them (#10391) @joseph-isaacs
• Support filters with limit in ScanBuilder (#8766) @AdamGS
• Add more scalar literal types to the Python and JNI bindings (#10079) @robert3005
• feat(array): support fill null for UTF-8 and binary arrays (#10044) @1fanwang
• feat(python): support row indices in VortexFile.to_arrow (#10373) @joseph-isaacs

🚀 Performance

• Plan V2 random access from known row selections (#10625) @joseph-isaacs
• perf(fastlanes): answer BitPacked compare and between from the bit width, between through the fused kernel (#10559) @mightsleep
• perf(array): snapshot aggregate kernels in ExecutionCtx (#10561) @joseph-isaacs
• Decode short RunEnd runs with fixed-size block writes (#10488) @joseph-isaacs
• Size the list layout split grid by row count (#10531) @mhk197
• Preserve contiguous index ranges in BitPacked take (#10520) @blaginin
• perf(fsst): gather sparse takes from small arrays with large payloads (#10509) @blaginin
• Deduplicate execute is_done check with done_canonical check (#10387) @robert3005
• perf(datafusion): avoid Arrow session lookup for ordinary scalars (#10541) @joseph-isaacs
• perf: make FieldSelection normalization linear (#10517) @joseph-isaacs
• perf: read pass-through struct projections directly (#10515) @joseph-isaacs
• perf: move decoded stats into arrays without stats (#10516) @joseph-isaacs
• Optimise bitpacked filtering and take for valid runs (#10281) @robert3005
• fix(datetime-parts): keep null rows null and order by parts (#10145) @robert3005
• perf: exit compute_referenced_values_mask early once every value is referenced (#10423) @joseph-isaacs
• Decode nullable RunEnd validity by clearing null streaks (#10487) @joseph-isaacs
• Drop DuckDB file scan state outside the multi-file lock (#10486) @robert3005
• perf(layout): skip chunks without selected rows (#10491) @robert3005
• Bound DashMap shard counts for per-reader caches (#10485) @robert3005
• Blocked BitPacked: fuse FoR decode over blocked children (#10251) @mhk197
• perf(mask): read intersect_by_rank rank bits at an absolute offset (#10480) @mightsleep
• avx2 branch for bool take kernel (#10385) @myrrc
• perf: enable POPCNT in the BMI2, AVX2 and AVX-512 kernels (#10456) @mightsleep
• feat(compressor): measure inputs and blocks by the bytes their rows reference (#10445) @joseph-isaacs
• perf(mask): SVE2 BDEP and a branch-free portable deposit for intersect_by_rank (#10168) @mightsleep
• feat(array): add precision option to uncompressed_size_in_bytes (#10439) @joseph-isaacs
• perf(layout): lower zoned pruning predicates once (#10428) @robert3005
• Optimize SearchSorted to reuse probes and avoid repeated context creation (#9904) @joseph-isaacs
• Speed up scans with a roaring row selection (#10359) @joseph-isaacs
• perf: batch fragmented CASE branch execution (#9820) @gatesn

🐛 Bug Fixes

37 changes

• fix: consult segment cache before registering reads (#10535) @gatesn
• Don't pass extension arrays through the compressor unchanged (#10424) @robert3005
• Validate MaskedArray with an explicit execution context (#10591) @robert3005
• fix: avoid fill_null crashes on lazy fill values (#10576) @mvanhorn
• chore: fix intermittent failure in fused_decode_is_wrapping_add (#10544) @mightsleep
• Fix constant decimal array overflow for decimals bigger than i128 (#10459) @robert3005
• fix: account for NaN in float stats exported to DuckDB and DataFusion (#10479) @jackylee-ch
• fix(datetime-parts): keep constant nullability in comparison pushdown (#10512) @robert3005
• Use SQL null semantics for DuckDB IN and NOT IN pushdown (#10054) @robert3005
• fix(datafusion): infer singleton distinct counts from exact bounds (#10448) @joseph-isaacs
• fix(python): translate Polars struct field access (#10194) @danking
• Struct reader partitioning respects struct validity (#10449) @myrrc
• Update rustls to 0.23.46 (#10492) @palaska
• fix: omit zone-map stats that a nullable struct's nulls would corrupt (#10413) @joseph-isaacs
• Fix DataFusion expression conversion and pushdown semantics (#9785) @AdamGS
• fix(array): scale decimal-to-float scalar casts across the whole scale domain (#9883) @jackylee-ch
• Only gather sparse takes from OnPair and FSST (#10421) @joseph-isaacs
• fix(layout): stop dict writer looping on an oversized value (#10400) @vortex-claude[bot]
• fix(file): validate metadata segments against the file size too (#9881) @jackylee-ch
• fix(python): support Polars datetime literals and timezone expressions (#10183) @danking
• fix: report a canonical NaN for a float sum, not the platform's default (#10122) @gerchowl
• fix(python): translate Polars fill_null with expression fills (#10192) @danking
• feat(expr): add timestamp timezone replacement (#10182) @danking
• fix(python): cast scan indices to u64 in read_url and to_array (#10277) @jackylee-ch
• fix(cloud): parse HF ALLOW_HTTP the way object_store does (#10278) @jackylee-ch
• Fix nullability of bool take (#10411) @robert3005
• Expression optimisation rules don't introduce irreducible children (#9625) @robert3005
• fix(python): translate Polars boolean NOT expressions (#10190) @danking
• fix(python): translate Polars is_null expressions (#10189) @danking
• fix(python): translate Polars binary scalar literals (#10188) @danking
• fix(python): translate Polars date literals (#10185) @danking
• fix(python): support Polars is_not_null predicates (#10184) @danking
• fix(python): support unsigned Polars scalar literals (#10170) @danking
• fix(python): translate Polars is_between expressions (#10191) @danking
• fix(python): translate Polars decimal literals (#10187) @danking
• fix(python): translate Polars conditional expressions (#10193) @danking
• Canonical dict execute handles edge cases in Decimal/Bool logic (#10098) @robert3005

🧰 Maintenance

34 changes

• GetItem reduction rule produces new_constant instead of new_node(Literal) (#10606) @robert3005
• compress-bench: write and scan Parquet the way DataFusion does by default (#10571) @robert3005
• rust toolchain 1.99.1 (#10608) @blaginin
• Validate RunEnd arrays without the legacy session (#10587) @robert3005
• Make Null take an execute kernel (#10585) @robert3005
• Lock file maintenance (#10598) @renovate[bot]
• Fix the manual_map lint in TrustedLenAdapter::next (#10597) @robert3005
• Use mimalloc with purging disabled in each e2e benchmark binary (#10490) @joseph-isaacs
• Audit tests for duplication and coverage (#10458) @robert3005
• Reuse unpack helpers in BitPacked range take (#10565) @robert3005
• ci(compress-bench): give Parquet and Vortex the same parallel CPU budget (#10510) @joseph-isaacs
• Write the list layout when the enabled editions include it (#10563) @mhk197
• Make the list layout opt-in and edition-gated (#10529) @mhk197
• Rewrite range pairs into between once per scan, and push down DataFusion dynamic filters (#10505) @robert3005
• bench: generate TPC-DS with tpcdsgen and share Parquet writer settings with TPC-H (#10511) @joseph-isaacs
• ci(python): run Python benchmarks with CodSpeed (#10500) @robert3005
• Use mimalloc as the global allocator in every benchmark binary (#10483) @joseph-isaacs
• Strengthen test assertions and remove fuzz reference checks (#10465) @robert3005
• Blocked BitPacked: add fastlanes.bitpacked.v2 serialization (#10244) @mhk197
• [bench] sparse bitpacked filter and take (#9900) @robert3005
• Update lance benchmark dependencies (major) (#10436) @renovate[bot]
• Update gradle/actions digest to 3f5f9ad (#10434) @renovate[bot]
• Update CodSpeedHQ/action digest to f842b9a (#10432) @renovate[bot]
• Update dtolnay/rust-toolchain digest to 686976e (#10433) @renovate[bot]
• Lock file maintenance (#10438) @renovate[bot]
• Update taiki-e/install-action digest to 1168f68 (#10435) @renovate[bot]
• Update Rust crate zip to v9 (#10437) @renovate[bot]
• Update actions/upload-artifact digest to cf430e0 (#10431) @renovate[bot]
• Update actions/setup-node digest to 949feb2 - autoclosed (#10430) @renovate[bot]
• Update actions/download-artifact digest to 9000827 (#10429) @renovate[bot]
• Return owned results from the optimizer and log every parent rewrite (#10407) @robert3005
• Expose slots, dtype, len and is_empty on TypedArrayRef (#10406) @robert3005
• Don't require boxing ChunkedArray chunk iterators (#9918) @robert3005
• Cache aggregate results on the requested array, not its canonical form (#10380) @robert3005