0.77.0

Changes

NOTE: 0.76.0 was yanked, because of a (hopefully very rare) latent correctness bug, that became much easier to hit in 0.76.0. There was a long-standing incorrect behavior in the TableStrategy writer, wherein it would not correctly propagate panics from the parallel writer subtasks. The fix was in #8672.

If you passed a stream of well-formed valid arrays straight to the builtin vortex writers, you are definitely unaffected.

If any of the following are true, you may have files that were silently truncated, in part (all versions) or in whole (0.76.0):

• you had a stream of well-formed arrays, AND called map on that ArrayStream with a function that panicked on invocation, then passed the resulting stream to a builtin vortex writer
• you passed a stream of malformed arrays that could have triggered a panic or assertion failure (e.g., you constructed invalid vortex arrays using unsafe/new_unchecked; to our knowledge you can't hit this path with the safe/regular constructors)

If you believe you may have been affected by this issue, please reach out in slack: https://vortex.dev/slack

⚠️ Breaks

• FFI: replace vx_string/vx_binary with vx_view (#8668) @myrrc
• Replace ToCanonical trait usages with execute (#8609) @robert3005
• Remove borrowed types from FFI; fixed_size_list constructor accepts u32 (#8658) @myrrc
• Remove variants module (#8643) @robert3005
• Disallow the legacy global session and allowlist existing call sites (#8637) @robert3005

✨ Features

• feat(geo): add native LineString, MultiPoint, MultiLineString types (#8679) @HarukiMoriarty
• Cast pushdown for duckdb (#8620) @myrrc
• mean() for decimals (#8649) @myrrc
• FFI: canonicalize array, data pointer for primitives (#8665) @myrrc
• VortexSession register is public and allow_unknown takes reference (#8647) @robert3005
• feat: support ST_DWithin pushdown in vortex (#8625) @HarukiMoriarty
• feat(vortex-bench): support vortex native geo types into SpatialBench, and wire into benchmark orchestrator (#8623) @HarukiMoriarty
• feat(vortex-geo): add MultiPolygon extension type + WKB export for native geometry (#8608) @HarukiMoriarty
• feat(vortex-bench): wire SpatialBench into the bench orchestrator (#8607) @HarukiMoriarty

🚀 Performance

• Optimize bit iteration and validity checking with SIMD popcount (#8636) @robert3005
• perf: branchless mask-select for listview zip (#8264) @joseph-isaacs

🐛 Bug Fixes

• fix: TableStrategy currently hides panic in spawned column writer tasks. (#8672) @AdamGS
• Grow list builders as we append elements to it (#8673) @robert3005
• Fix vortex-jni local path handling, centralise logic in vortex-file (#8650) @robert3005

🧰 Maintenance

13 changes

• Consolidate benchmark SQL queries into vortex-bench/sqldirectory (#8684) @connortsui20
• Direct issues to templates instead of GitHub Discussions (#8685) @connortsui20
• Better error message in case of unknown encoding id (#8681) @robert3005
• Hardcode constant detection into the cascading compressor (#8667) @connortsui20
• Add policy-bot configuration file (#8659) @robert3005
• Lock file maintenance (#8654) @renovate[bot]
• Add bit iteration benchmarks (#8656) @robert3005
• Lock file maintenance (#8655) @renovate[bot]
• Update Rust crate cxx to v1.0.195 [SECURITY] (#8653) @renovate[bot]
• bench: warm ArrayKernels before the bench loop (#8479) @joseph-isaacs
• Remove execute_boolean fallback to arrow (#8644) @robert3005
• Decouple optional features from pulling vortex-file features for top crate (#8642) @AdamGS
• feat: add CUDA cuDF convenience API (#8624) @0ax1