v0.38.0

What's Changed

🎉 🎉 ❗ As of this release, the 2.1 version of the file format is considered stable. There will be no more breaking changes and the format should be readable by future versions of lance. In an upcoming release (possibly the next release) the 2.1 version will become the default.

Breaking Changes 🛠

• feat(rust)!: support branch based on shallow_clone by @majin1102 in #4710

New Features 🎉

• feat: support manifeset summary and get it from Version by @majin1102 in #4754
• feat: implement blob encoding for 2.1 by @westonpace in #4802
• feat(java): expose additional Operation::Update fields to bindings by @wayneli-vt in #4788
• feat(index/fts): build fst at write time instead of read time by @Xuanwo in #4811
• feat: support RabitQ quantization by @BubbleCal in #4344
• feat: add scalar and system index spec by @jackye1995 in #4736
• feat: add support for already-dictionary to 2.1 by @westonpace in #4813
• feat: allow reading blob descs in 2.1, fix bugs in FSL decoder when items are nullable by @westonpace in #4840
• feat: add full text json index by @wojiaodoubao in #4752

Bug Fixes 🐛

• fix: propagate span in filtered read by @wjones127 in #4790
• fix: add incremental metrics in FilteredRead by @wjones127 in #4798
• fix: fix handling of all-preamble chunks in mini-block scheduling by @westonpace in #4823
• fix: don't interpret 64-bit offsets as 32 bits when decompressing per-value data in 2.1 by @westonpace in #4824
• fix: aggregate gauge metrics in bytes_read for zonemap scans by @chenghao-guo in #4830
• fix: remove blocking call in async function by @wjones127 in #4841
• fix: index stats reports incorrect partition size by @BubbleCal in #4847
• fix: fix typo "blfoat16" -> "bfloat16" in datatypes.rs by @huyuanfeng2018 in #4852
• fix: blocking_take may hang on V2.0 by @ColdL in #4539
• fix: schema mismatch when filtering nullable List columns by @Xuanwo in #4838
• fix: ensure fragment IDs are sorted in load_training_data by @jtuglu1 in #4871

Documentation 📚

• docs: vector index specs by @BubbleCal in #4810

Performance Improvements 🚀

• perf: move cpu heavy fst building into blocking threads by @Xuanwo in #4803
• perf: optmize doc set loading by @Xuanwo in #4821

Other Changes

• refactor: cowardly refuse to use scalar indexes on expressions with null literals by @westonpace in #4815

New Contributors

• @wayneli-vt made their first contribution in #4788
• @huyuanfeng2018 made their first contribution in #4852
• @ColdL made their first contribution in #4539
• @jtuglu1 made their first contribution in #4871

Full Changelog: v0.37.0...v0.38.0