v0.37.0

What's Changed

Breaking Changes 🛠

• feat: add table metadata, support incremental updates of all metadata by @wjones127 in #4350

New Features 🎉

• feat: optimize bitmap index with lazy loading and column projection by @LuQQiu in #4699
• feat: add LanceFileSession.open_writer for sharing object store by @jmhsieh in #4722
• feat: apply bitpacking on rep/def for compression by @Xuanwo in #4537
• feat: support create btree index distributedly by @xloya in #4667
• feat: add LANCE_LOG_FILE environment variable for file logging by @yanghua in #4721
• feat: optimize file name for s3 throughput by @jackye1995 in #4737
• feat(rust): support refresh frag bitmap for index after updating when… by @yanghua in #4589
• feat(java): supports do compaction distributedly by @steFaiz in #4706
• feat: exposing option to provide a serialized manifest to the Java SDK by @morales-t-netflix in #4764
• feat: allow out-of-line bitpacking supports tail chunk by @Xuanwo in #4740
• feat: add udtf for fts query by @wojiaodoubao in #4684
• feat: support nested field for index operations by @jackye1995 in #4682

Bug Fixes 🐛

• fix: refine SIMD support detection for AArch64 on iOS/tvOS by @felix-schultz in #4725
• fix: fix various 2.1 writer issues by @westonpace in #4713
• fix: correct underestimate of Bloom filter epsilon by @jbapple in #4734
• fix: fix the preview release is wrong for preview versions by @Xuanwo in #4750
• fix(rust): fix rechunk sequences bug and refactor for better readability by @yanghua in #4695
• fix(FlatMatchQuery): add support for List of Utf8 by @wojiaodoubao in #4742
• fix: fts boolean query and boost query do not support phrase by @wojiaodoubao in #4766
• fix: speed up generating ProjectionPlan for full schema by @timsaucer in #4743
• fix: don't create an empty chunk when values are evenly divisible by @westonpace in #4775
• fix: fix decoder bugs in 2.1 list handling, expand test coverage by @westonpace in #4784
• fix: move indexes back to table.proto to avoid forward compat. issues by @westonpace in #4786

Documentation 📚

• docs: update performance.md for index cache section by @chenghao-guo in #4738
• docs: make duckdb example runnable by @eddyxu in #4770

Performance Improvements 🚀

• perf: avoid per-iop clone of filename by @westonpace in #4714
• perf: hierarchical clustering by @BubbleCal in #4726
• perf: cache num_cpus::get by @westonpace in #4768

Other Changes

• refactor: rename lance_table::format::Index to lance_table::format::IndexMetadata by @westonpace in #4760
• refactor(rust): distinguish row id and row addr part-1 by @yanghua in #4352

New Contributors

• @jmhsieh made their first contribution in #4722
• @felix-schultz made their first contribution in #4725
• @xloya made their first contribution in #4667
• @morales-t-netflix made their first contribution in #4764

Full Changelog: v0.36.0...v0.37.0