v0.21.0

What's Changed

Breaking Changes 🛠

• fix!: correctly handle nulls in btree and bitmap indices by @westonpace in #3211
• feat!: support hamming distance & binary vector by @BubbleCal in #3198
• refactor(python)!: simplify marshalling of Fragment, DataFile, Operation, Transaction by @wjones127 in #3240

New Features 🎉

• feat: enhance repdef utilities to handle empty / null lists by @westonpace in #3200
• feat: support _rowid meta column for spark connector in java by @SaintBacchus in #3194
• feat: support blob api in pytorch loader by @eddyxu in #3217
• feat(python): add experimental parameter enable_move_stable_row_ids for pylance by @SaintBacchus in #3216
• feat: add the repetition index to the miniblock write path by @westonpace in #3208
• feat: packed struct encoding by @broccoliSpicy in #3186
• feat: support between sql clauses by @connellPortrait in #3225
• feat(java): support drop columns for dataset by @yanghua in #3237
• feat(java): expose uri method for Dataset instance by @yanghua in #3231
• feat: add file statistics by @broccoliSpicy in #3232
• feat: enable tracing for object storage by @wjones127 in #3244
• feat(java): support limit and offset interface for spark connector by @SaintBacchus in #3253
• feat: adds list decode support for mini-block encoded data by @westonpace in #3241
• feat(java): support topn pushdown in spark connector by @SaintBacchus in #3261
• feat: add replace_schema_metadata and replace_field_metadata by @westonpace in #3263
• feat: merge-insert supports inserting subset of columns by @wjones127 in #3100
• feat: support merge by row_id, row_addr by @chenkovsky in #3254
• feat: add the s3 retry config options for storage option by @SaintBacchus in #3268
• feat(java): support alter columns for dataset by @yanghua in #3259
• feat: support remapping for IVF_FLAT, IVF_PQ and IVF_SQ by @BubbleCal in #2708
• feat: change MSRV from 1.78 to 1.80.1 by @westonpace in #3279
• feat: support merge fragment with dataset by @chenkovsky in #3256

Bug Fixes 🐛

• fix: test failure in test_fsl_packed_struct by @broccoliSpicy in #3227
• fix: remove overzealous warning by @westonpace in #3239
• fix: correctly copy null buffer when making deep copy by @westonpace in #3238
• fix: allow LANCE_LOG to be set to trace by @westonpace in #3246
• fix: list indices always shows vector index type is IVF_PQ even it's not by @BubbleCal in #3258
• fix: panic when get stats from index over binary vectors by @BubbleCal in #3267
• fix(rust): adjust scan range to avoid unnecessary warnings by @takaebato in #3248
• fix: when taking struct fields they should be merged into the output in the correct order by @westonpace in #3277
• fix: full text search with limit may return an incorrect results by @BubbleCal in #3284
• fix: refine type annotation by @chenkovsky in #3278

Documentation 📚

• docs: add the documentation about how to install packages for tests by @yanghua in #3213
• docs: add doc and test for 4bit PQ by @BubbleCal in #3212
• docs: blob api documents by @eddyxu in #3247
• docs: add java module into directory structure by @yanghua in #3273

Performance Improvements 🚀

• perf: in-register lookup table & SIMD for 4bit PQ by @BubbleCal in #3178

New Contributors

• @connellPortrait made their first contribution in #3225
• @takaebato made their first contribution in #3248

Full Changelog: v0.20.0...v0.21.0