v1.0.0-beta.15

What's Changed

Breaking Changes 🛠

• perf!: dynamic pruning for vector search by @BubbleCal in #4773
• feat!: remove unnecessary mut of dataset::sql by @ddupg in #5207
• refactor!: move all previous code into previous mod by @Xuanwo in #5217
• refactor!: deprecate TFRecord support by @jackye1995 in #4593
• refactor!: use org.lance namespace for java package by @jackye1995 in #5339

Critical Fixes ‼️

• fix: convert some panics into errors by @wjones127 in #5258

New Features 🎉

• feat: provide inline_transaction model for IO optimizing by @majin1102 in #4774
• feat(python): add support for HuggingFace IterableDataset by @changhiskhan in #2599
• feat: add public accessors for count plan construction by @wkalt in #5103
• feat: add adapter for REST namespace with manifest namespace backend by @jackye1995 in #4984
• feat: add blob compaction support by @Xuanwo in #5189
• feat: add inline optimization for dir namespace by @jackye1995 in #5244
• feat: conflict resolution for DataReplacement by @wjones127 in #3631
• feat: support namespace vended credentials for write by @jackye1995 in #5161
• docs: correct the comment in util.py by @shiyajuan123 in #5252
• feat: introduce community governance by @jackye1995 in #5262
• feat: add describe_indices function by @westonpace in #5221
• feat: add target_bases extension to python write_fragments API by @jaystarshot in #5234
• feat: support credentials vending for file reader and session by @jackye1995 in #5256
• feat: introduce blob arrow extension type by @Xuanwo in #5239
• feat(java): add binding for rest and dir namespaces by @jackye1995 in #5292
• feat: expose file upload and download in Lance file session by @jackye1995 in #5336
• feat(java): support credential vending at write time by @jackye1995 in #5309
• feat(ds-sql-api): support JSON bulit in functions in ds.sql API by @zhangyue19921010 in #5350
• feat: support GEO types by @ddupg in #4678
• feat(python): expose DatasetDeltaBuilder and relevant apis by @yanghua in #5091
• feat: return Unprocessable error while expected error happened by @Xuanwo in #5347
• feat: add huggingface native support by @Xuanwo in #5353
• feat: dynamically choose distance type by @BubbleCal in #5370

Bug Fixes 🐛

• fix: ensure recheck for IsNotNull in bloom filter by @Xuanwo in #5192
• fix: contributing URL gives 404 by @prrao87 in #5196
• fix: merge struct array use wrong child values by @wojiaodoubao in #5106
• fix: avoid unnecessary get_fragments calling during plan compaction by @zhangyue19921010 in #5179
• fix: split partition may be assigned to itself by @BubbleCal in #5190
• fix: improve schema validation for nullability and subschemas by @fenfeng9 in #4994
• fix: compile error in test_inline_transaction by @wojiaodoubao in #5206
• docs: fix batch udf with checkpoint document error by @fangyinc in #5185
• fix: memory-limited string BTREE index building by @wjones127 in #5175
• fix: home page code snippets cuasing various problems by @jackye1995 in #5245
• fix: panic if only one partition and split is triggered by @BubbleCal in #5241
• fix: clearer error in dataset take by @wkalt in #5243
• fix: docs and comment have broken links by @prrao87 in #5261
• fix: handle logical rows deletion properly for zonemap and bloomfilter by @HaochengLIU in #5140
• fix: blob version should be passed in Projection by @Xuanwo in #5295
• docs: fix broken links and 404s by @prrao87 in #5284
• fix: index overestimates the posting list size by @BubbleCal in #5327
• fix: update CachedFileMetadata version API to V2.0 by @zhangyue19921010 in #5330
• fix: correctly handle OSS commit protocol to prevent data loss by @Pmathsun in #5332
• fix: update btree index with its own zone size instead of DEFAULT_BTR… by @ztorchan in #5301
• fix: join job may cause inconsistent delta indices by @BubbleCal in #5328
• fix: panicException when calling compaction by @yanghua in #5282
• fix: parallelize bitmap partition loading in IsIn expressions by @wkalt in #5355

Documentation 📚

• docs: introduce lance as a lakehouse format by @jackye1995 in #5209
• docs: update readme with latest lakehouse format info by @jackye1995 in #5216
• docs: capitalization change by @timsaucer in #5269
• docs: build project specific guidelines into web doc by @jackye1995 in #5324

Performance Improvements 🚀

• perf: speed up filtered scan by up to 18.9× by moving the heavy CPU task out by @Xuanwo in #5165
• perf: parallelize split job assigning by @BubbleCal in #5265
• perf: use CPU pool to run WAND algo by @BubbleCal in #5363

Other Changes

• refactor: move blob version as a table level config by @Xuanwo in #5220
• refactor: add helper functions to delta.rs tests by @yanghua in #5298
• refactor: move LanceNamespace interface to pylance and java lance-core by @jackye1995 in #5345
• refactor: allow datafiles to contain columns without field id by @Xuanwo in #5348
• refactor: rename RowIdSelection to RowAddrSelection by @yanghua in #5263
• refactor: separate out python and java LanceNamespace interface by @jackye1995 in #5364
• refactor: align with blob v2 logical types change by @Xuanwo in #5375

New Contributors

• @prrao87 made their first contribution in #5196
• @fenfeng9 made their first contribution in #4994
• @fangyinc made their first contribution in #5185
• @shiyajuan123 made their first contribution in #5252
• @Pmathsun made their first contribution in #5332
• @ztorchan made their first contribution in #5301
• @fMeow made their first contribution in #5371

Full Changelog: release-root/1.0.0-beta.N...v1.0.0-beta.15