0.84.0
·
150 commits
to develop
since this release
Changes
⚠️ Breaks
- refactor: preserve bound expression identity through scans (#9192) @joseph-isaacs
- FFI: get scalars from arrays (#8896) @myrrc
- refactor: support formatting of bound expressions (#9228) @joseph-isaacs
- Add a Normalized encoding for tensor columns (#9138) @connortsui20
- refactor: propagate bound expressions through layouts (#9126) @joseph-isaacs
- Replace DynVarBinBuilder with a generic VarBinBuilder (#9096) @robert3005
- Constrain List/ListView builder offset, size types to 32 and 64 bit types (#9053) @robert3005
- Replace LayoutStrategy buffered_bytes with a buffered bytes tracker (#8961) @robert3005
- Route StructArray through the array_slots accessors and drop unmasked_fields() (#9006) @joseph-isaacs
- Refactor layout child access to support optional slots (#9000) @joseph-isaacs
- FFI: remove arc/arc_dyn wrapper (#9002) @myrrc
- Rename scalar function strictness contract (#8930) @connortsui20
- Pin every
#[array_slots]field to an explicit slot index (#9004) @joseph-isaacs - Replace all usages of Registry with ArcSwapMap (#8919) @robert3005
- chore: Inline slot index aliases into generated slot constants (#9003) @joseph-isaacs
- refactor(array): own struct field vectors (#8957) @joseph-isaacs
✨ Features
- feat: implement take for canonical sparse Union arrays (#9246) @connortsui20
- Read hf:// URLs from DuckDB by mounting stores where the registry says (#9265) @robert3005
- take() on VarBin returns VarBinView (#9229) @myrrc
- Push hugging face storage resolution into vortex-cloud rather than vortex-python (#9199) @robert3005
DecimalMul/Div kernels (#9221) @mhk197- feat: add OpenDAL-backed GooseFS object store support (#9134) @XuQianJin-Stars
- Use RowIdxLayoutReader only when #row_idx is referenced (#9190) @myrrc
- Support Map type through JNI and Spark (#9209) @AdamGS
- Share fixed-width structural array operations (#8864) @connortsui20
- Add Map core edition (#9195) @AdamGS
- Add Arrow Map roundtrip support (#9111) @AdamGS
- Add Map compute and compression support (#9110) @AdamGS
- feat(datafusion): allow configuring
ExpressionConvertoronVortexFormatandVortexFormatFactory(#9185) @haohuaijin - Add canonical Map arrays and builder (#9108) @AdamGS
- Add Map logical type and scalar support (#9107) @AdamGS
- feat(expr): add bound expressions (#9125) @joseph-isaacs
- Exectuion and optimization tracing harness for testing (#7814) @AdamGS
- Revert "Derive optimized ScanPlanV2s from source plans" (#9124) @joseph-isaacs
- SLT tests for aggregations (#9086) @myrrc
- refactor(cloud): split cloud object store integration into a vortex-cloud crate (#8999) @robert3005
- Derive optimized ScanPlanV2s from source plans (#9005) @joseph-isaacs
- feat[file]: gate the writer on the session's enabled editions (#8946) @joseph-isaacs
- Add support for huggingface datasets to vortex-python (#8733) @robert3005
- feat: add a ci gpu compress benchmark (#8951) @joseph-isaacs
- Expose metadata reading and writing in vortex-jni (#8963) @robert3005
- Support variadic tail slots in #[array_slots] macro (#8950) @joseph-isaacs
- feat(vortex-python): Add OpenDAL-backed CosStore to the Python object-store API (#8845) @XuQianJin-Stars
- feat(arrow): export StringArray and BinaryArray directly (#8902) @ClSlaid
- feat[edition]: add enabled-editions selection to the session (#8945) @joseph-isaacs
🚀 Performance
- Skip materializing splitless chunk children during split collection (#9271) @AdamGS
- Improve take on Chunked(FSL) and FSL(Chunked) array alongside implementation of take using PiecewiseSequence (#8881) @robert3005
- Lazily untranspose DeltaArray validity take #2 (#9226) @robert3005
- Optimize ChunkedArray take routing (#9224) @gatesn
- Optimize split calculation in DF source (#9167) @AdamGS
- Vectorize checked multiplication by reporting evidence, not a comparison (#9210) @connortsui20
- Add scalar function performance baselines (#9136) @connortsui20
- duckdb: don't use tokio for local reads (#9175) @myrrc
- Shrink zstd buffer before returning (#9179) @halvorboe
- perf(vortex-geo): short-circuit predicates with bounding boxes (#9076) @HarukiMoriarty
- perf(onpair): adopt onpair 0.2 storage-backed dictionary and safety-only validation (#9117) @gargiulofrancesco
- perf(array): stop flattening appended list views in ListViewBuilder [builders-child-stack] (#9046) @claude[bot]
- block_on_stream_thread_safe: increase channel size to number of cores (#9116) @myrrc
- Use unchecked ListViewArray construction after validation (#9044) @robert3005
- optimize the array creation (slots) (#9013) @joseph-isaacs
- Unify numeric compute kernels with vortex-compute iteration functions (#8939) @robert3005
- Execute until arrays match fast path or canonical before exporting to Arrow (#8935) @mhk197
- Back EnabledEditions with a session Registry (#8956) @joseph-isaacs
- feat(expr): fold cast of a literal to a literal in Cast simplify (#8962) @HarukiMoriarty
- Enable metadata-only mask reduction for constant Boolean masks (#8934) @connortsui20
🐛 Bug Fixes
30 changes
- duckdb: don't panic on unsupported Value type (#9243) @myrrc
- duckdb: Restrict pushed aggregations (#9233) @myrrc
- Fix decimal scalar multiplication and division (#8888) @mhk197
- duckdb: convert aggregate results to duckdb requested output type (#9231) @myrrc
- Validate row-index selections (#8574) @gatesn
- duckdb: TIMESTAMPTZ keeps microsecond precision (#9241) @myrrc
- duckdb: don't return stats() for aggregate scans and scans with pushed projection expression (#9230) @myrrc
- Fix .take() overflow on VarBin with >4GB data (#9227) @myrrc
- Fill in missing functionality for maps in fuzzer (#9208) @AdamGS
- Remove the unimplemented
NotKernelextension point (#9214) @connortsui20 - fix(buffer): bound alignment exponents read from untrusted input (#9155) @lorenzhs
- Panic on DType::Null conversion to NonNullable (#9207) @myrrc
- Upgrade to fsst 0.6.0 and guard against out of bounds codes (#9149) @robert3005
- fix(spark): write null array elements as nulls instead of zeros (#9169) @jackylee-ch
- fix(spark): delete aborted output through the native filesystem layer (#9172) @jackylee-ch
- Use all cores for parquet->vortex conversion in duckdb-bench (#9165) @myrrc
- fix(zstd): validate frame content sizes (#9164) @lorenzhs
- fix
get_itemparent reduce rule (#9151) @mhk197 - Harden RLE decompression against untrusted indices and offsets (#8953) @joseph-isaacs
- Reenable float aggregation pushdown for duckdb except for min() (#9094) @myrrc
- fix: validate unsigned AVX2 take indices (#9036) @connortsui20
- mean(all null) = NULL (#9079) @myrrc
- fix: LazyScanStream prepare/execute is CPU-work not I/O (#9078) @robert3005
- Mark file-level min/max utf8/binary stats exactness correctly (#9088) @AdamGS
- duckdb: don't push down aggregation over floats (#9069) @myrrc
- make alp-rd dicts deterministic on equal frequency entries (#9032) @onursatici
- Fix escaped field names in calculate_physical_schema (#9049) @brancz
- fix: Fix datafusion benchmark, attaching table ref to table URL (#9014) @AdamGS
- fix(cuda): pass mutated CUDA device buffers as
mut(#9001) @0ax1 - fix(vortex-row): decimal sort keys are not memcmp-comparable across chunks (#8937) @HarukiMoriarty
📖 Documentation
6 changes
is_strictandis_fallibledocs improvement (#9122) @connortsui20- docs: make the 1 ms benchmark iteration budget an explicit maximum (#9176) @joseph-isaacs
- Adjust strict docs (#9033) @connortsui20
- docs: Remove empty developer guide pages (#9026) @myrrc
- Smaller docs website (#8942) @myrrc
- docs: clarify vortex-row is not production-ready (#8949) @joseph-isaacs
🧰 Maintenance
47 changes
- VarBinViewBuilder: support decoding Dict and Sparse arrays directly into the builder (#9236) @robert3005
- VarBinViewBuilder: support appending buffers directly into the builder storage (#9235) @robert3005
- Collapse the checked-arithmetic macros into one integer body (#9248) @connortsui20
- Improve agents related toolings/docs (#9264) @AdamGS
- test(spark): characterize VortexScanBuilder predicate pushdown splitting (#8857) @jackylee-ch
- VarBin: build views straight into a VarBinViewBuilder (#9127) @robert3005
- OnPair: decode straight into the builder's byte storage (#9100) @robert3005
- Zstd: append values in bulk and harden the frame metadata reads (#9099) @robert3005
- Remove stale strategy override in vortex-jni (#9102) @robert3005
- Dict: gather dictionary values straight into a VarBinBuilder (#9098) @robert3005
- FSST: decompress straight into the builder's byte storage (#9097) @robert3005
- chore(deps): bump opendal to 0.58.1 (#9163) @Xuanwo
- Fix DF benchmark pushdown options (#9184) @mprammer
- duckdb: more const for FFI (#9173) @myrrc
- docs(spark): update the published version coordinates (#9171) @jackylee-ch
- test(spark): characterize VortexCatalog path identifier resolution (#9161) @jackylee-ch
- test(spark): characterize VortexTable capabilities and builders (#9160) @jackylee-ch
- fix(spark): correct the @throws contract on VortexScanBuilder.build (#9162) @jackylee-ch
- chore: tidy up a few compile-time assertions (#9158) @connortsui20
- fix(spark): include the offending path when schema inference finds no Vortex files (#8858) @jackylee-ch
- Lock file maintenance (#9140) @renovate[bot]
- test(spark): characterize HadoopUtils S3 and Azure credential forwarding (#8850) @jackylee-ch
- test(spark): characterize VortexBatchExec input partition planning (#8859) @jackylee-ch
- Resolve semantic cargo lock conflict (#9145) @robert3005
- Use upstream ALP crate instead of our own implementation (#9103) @robert3005
- benchmarks: add string compression benchmark (#9060) @gargiulofrancesco
- fix(arrow): return an error for incompatible Arrow export targets (#8993) @robert3005
- Download duckdb sources only once per config/version combo (#9031) @AdamGS
- Purge remnants of the old benchmarks website versions (#9015) @connortsui20
- fix(deps): update lance benchmark dependencies (major) (#8989) @renovate[bot]
- chore(deps): lock file maintenance (#8991) @renovate[bot]
- chore(deps): lock file maintenance (#8992) @renovate[bot]
- chore(deps): update actions/setup-node action to v7 - autoclosed (#8982) @renovate[bot]
- chore(deps): update actions/checkout digest to 3d3c42e (#8970) @renovate[bot]
- chore(deps): update actions/setup-python action to v7 (#8983) @renovate[bot]
- chore(deps): update taiki-e/install-action digest to 41049aa (#8980) @renovate[bot]
- chore(deps): update actions/setup-java digest to 03ad4de (#8971) @renovate[bot]
- chore(deps): update actions/setup-node digest to 2499707 (#8972) @renovate[bot]
- chore(deps): update anthropics/claude-code-action digest to be7b93b (#8973) @renovate[bot]
- chore(deps): update aws-actions/configure-aws-credentials digest to e6de054 (#8974) @renovate[bot]
- chore(deps): update dorny/paths-filter digest to 7b450ff (#8975) @renovate[bot]
- chore(deps): update actions/checkout action to v7 (#8981) @renovate[bot]
- chore(deps): update dtolnay/rust-toolchain digest to 4cda84d (#8976) @renovate[bot]
- chore(deps): update embarkstudios/cargo-deny-action digest to 3c63498 (#8977) @renovate[bot]
- chore(deps): update pypa/gh-action-pypi-publish digest to ba38be9 (#8978) @renovate[bot]
- chore(deps): update ssciwr/doxygen-install digest to b5b80f5 (#8979) @renovate[bot]
- chore(deps): update rust crate syn to v3 (#8988) @renovate[bot]