0.88.0

Important breaks

None

Changes

⚠️ Breaks

• Speed up scalar construction, stats reads and aggregate partial merging (#10326) @robert3005
• break(arrow): add export options and allow skipping view buffer compaction (#10220) @lorenzhs
• Blocked BitPacked: store block offsets as a child (#10004) @mhk197
• Delta validity is a top level child (#9970) @robert3005
• Move users of map_each_with_validity and map_each_in_place to lane kernels and remove those methods (#10139) @robert3005

🚧 Deprecation

• Remove usages of null_as_false (#10124) @myrrc

✨ Features

• Add support for reading vortex files through native python readers (#10097) @robert3005
• duckdb: don't execute validity in exporters (#10376) @myrrc
• Blocked BitPacked: encode blocked bitwidths (#10203) @mhk197
• AllFalse early exit (#10303) @myrrc
• Scan protocol traits (#10047) @joseph-isaacs
• Add combine_chunks option to Python Arrow export (#10279) @robert3005
• feat(python): support integer column indices in read_url projection (#10287) @jackylee-ch

🚀 Performance

• Bool take doesn't fill_null validity (#10383) @myrrc
• perf(array): expand FixedSizeList filter masks directly into a bitmap (#10274) @robert3005
• Make opening a Vortex file with a cached footer cheaper (#10358) @joseph-isaacs
• perf: classify canonical arrays by concrete vtable (#10238) @joseph-isaacs
• Add OnPair take kernel that shares the pair dictionary (#10363) @joseph-isaacs
• perf(arrow): preserve source nullability during export (#10365) @joseph-isaacs
• Read integer validity a word per chunk, and fill nulls for runs (#10342) @joseph-isaacs
• Read VarBin UTF-8 RowFn inputs through their offsets (#10340) @connortsui20
• Execute FSST iteratively (#10209) @joseph-isaacs
• perf(array): avoid errors when comparison kernels decline arithmetic (#10301) @connortsui20
• Faster ScalarFnArray creation, lazy ArrayStats (#10329) @myrrc
• perf(buffer): use fearless_simd for popcount (#10327) @joseph-isaacs
• Initialise ArrayParts directly with provided slots (#10072) @robert3005
• Execute OnPair iteratively (#10210) @joseph-isaacs
• perf(compressor): detect constant primitives before generating stats (#10043) @robert3005
• perf: vectorize small u8 table take with AVX2 (#9572) @joseph-isaacs
• perf: vectorize small u8 table take with NEON (#9571) @joseph-isaacs
• Optimise min/max computation for VarBinView arrays (#9740) @robert3005
• Execute DecimalByteParts iteratively (#10207) @joseph-isaacs
• Execute RunEnd iteratively (#10206) @joseph-isaacs
• perf(zstd): reuse one decompression context per thread (#10270) @joseph-isaacs
• feat: add between kernel for DecimalByteParts (#10152) @joseph-isaacs
• python: release the GIL only around expensive work (#10280) @robert3005
• Convert trivial filters into slices during reduction, MaskValues::last uses BitBuffer::last_set_index (#9831) @robert3005
• perf(array): forward probe_scalar through pass-through encodings (#9905) @joseph-isaacs
• Perform validation and null view replacement in two passes (#10148) @robert3005

🐛 Bug Fixes

13 changes

• fix(python): translate Polars time literals (#10186) @danking
• fix(buffer): require both halves of a Zip to be TrustedLen (#10293) @jianhe25
• fix(buffer): validate offsets in Buffer::into_arrow_offset_buffer (#10292) @jianhe25
• fix(buffer): don't create &mut [u64] over uninitialized memory in collect_words_in (#10294) @jianhe25
• Trim child data on Arrow slice imports (#10379) @palaska
• duckdb: i16 Decimal on duckdb side may fit in Vortex's i8 decimal buffer (#10155) @myrrc
• Keep the file sum absent when a chunk sum overflows (#10328) @connortsui20
• Ignore empty constant chunks in aggregate states (#10254) @connortsui20
• fix(parquet-variant): decline all_non_distinct when only one side is shredded (#10275) @jackylee-ch
• fix(python): preserve nullability when converting a chunked array with nulls (#10283) @jackylee-ch
• fix(buffer): enable and detect bmi1 for _tzcnt_u64 in select_in_word (#10295) @jianhe25
• fix(array): preserve result nullability when simplifying and/or/zip (#10286) @jackylee-ch
• fix(datafusion): preserve list element field metadata in physical schema (#10285) @jackylee-ch

🧰 Maintenance

14 changes

• Skip validation when accessing storage scalar of extension scalar (#9875) @robert3005
• Validate TPC-DS results against Vortex and Parquet (#9863) @joseph-isaacs
• Test IO file sizes in Rust and array sizes in Python doctests (#9792) @robert3005
• chore(array): use fearless_simd for AVX2 1- and 2-byte filter compress (#10324) @joseph-isaacs
• Add sum oracle to the fuzzer and reject samples that have order dependent overflow behaviour (#9801) @robert3005
• Do not build a Sequence array for slices that are not sequences (#10338) @robert3005
• perf(buffer): portable fearless_simd kernels for bitmap rank-select (#10319) @joseph-isaacs
• fix: reconcile asyncband references in Cargo.lock (#10306) @joseph-isaacs
• Fix deprecations flagged by cargo build on new rust version (#10042) @robert3005
• Add list_contains membership benchmarks (#10060) @robert3005
• Clean up vortex_ensure_eq! (#10237) @connortsui20
• Lock file maintenance (#10290) @renovate[bot]
• Split SQL PR benchmarks into bench-sql and bench-sql-extended (#10213) @joseph-isaacs
• Update bit-vec to 0.10.1 (#10288) @jianhe25