v2.0.0-beta.5

What's Changed

Breaking Changes 🛠

• fix!: null handling when using NOT with scalar indices by @wjones127 in #5270
• feat!: track cumulative wall time in analyze plan by @wkalt in #5505

New Features 🎉

• feat(blob_v2): add external blob support by @Xuanwo in #5385
• feat: fallback to CPU if GPU accelerating is unavailable by @BubbleCal in #5407
• feat: disable default features on internal use by @valkum in #5372
• feat(blob_v2): add dedicated blob support by @Xuanwo in #5406
• feat(blob_v2): add packed blob support by @Xuanwo in #5413
• feat: strategized plan compaction by @zhangyue19921010 in #5233
• feat: support add sub-column to struct col by @wojiaodoubao in #5126
• feat: distributed range-based BTree index by @steFaiz in #5202
• feat: upgrade lance-namespace to 0.3.1 and add missing apis by @jackye1995 in #5457
• feat: add additional index APIs to support count rows split plan by @jackye1995 in #5447
• feat(blob_v2): add BlobAray API for user input by @Xuanwo in #5451
• feat: support using FTS as a filter in vector search by @wojiaodoubao in #4928
• feat(java): support multi-bases for writing database by @ddupg in #5450
• feat: add py.typed marker file by @jonded94 in #5479
• feat: support map data type in lance format version 2.2 by @xloya in #5349
• feat(blob_v2): add GC support by @Xuanwo in #5473
• feat(python): support cleanup_with_policy by @ddupg in #5458
• feat(java): simplify the use of optional in jni by @ddupg in #5488
• feat(python): add DatasetBasePath stub to improve IDE hints by @ddupg in #5503
• feat: cleanup only scan managed files by @majin1102 in #5338
• feat(java): support row lineage and cdf apis by @yanghua in #5362
• feat(memtest): add macos support by @Xuanwo in #5510
• feat: add support for large minichunk size (u32) in format v2.2 by @niyue in #4959
• feat(blob_v2): add Python API for Blob v2 by @Xuanwo in #5491
• feat: support global tag retrieval and improve tag api by @majin1102 in #5088
• feat: add RTree index spec in table format by @ddupg in #5360
• feat: dataset supports deep_clone by @majin1102 in #5250

Bug Fixes 🐛

• fix: don't allow change blob version during update by @Xuanwo in #5386
• fix: respect index metric when user overrides by @BubbleCal in #5395
• fix: take_blobs_by_indices fails with stable row IDs on fragment 1+ by @jmhsieh in #5392
• fix: remove expensive clone in bitmap search by @westonpace in #5409
• fix: stop documenting FTS index type, standardize on INVERTED by @mackrorysd in #5315
• fix: remove logging for project_batch by @westonpace in #5267
• fix: fix vector index prewarm index by @xloya in #5412
• fix: panic unwrap on None in decoder.rs by @camilesing in #5424
• fix: dir namespace cloud storage path removes one subdir level by @jackye1995 in #5464
• fix: ensure trailing slash is normalized in rest adapter by @jackye1995 in #5499
• fix: head external manifest object happend 404 NotFound error by @hushengquan in #5512
• fix: infer multivector sampling rows by @BubbleCal in #5534
• fix: json's arrow extension metadata missing by @Xuanwo in #5527
• fix: support ManifestNamingSchemeV2 with unordered object stores by @wjones127 in #5539
• fix: allow storage options provider without expires_at_millis by @jackye1995 in #5542
• fix(ci): use pull_request_target for fork PR reviews by @wjones127 in #5544
• fix: make column name lookups case-insensitive by @wjones127 in #5465
• fix: merge_insert uses full schema path for reordered columns by @wjones127 in #5541
• fix: correct null_count aggregation in boolean statistics collection by @YinZheng-Sun in #4839
• fix: restore decrease max_fragment_id in manifest by @majin1102 in #5554

Documentation 📚

• docs: fix Append call in distributed write guide by @rongou in #5439
• docs: fix and improve the description about row id by @yanghua in #5463
• docs: fix duplicate words in comments and error messages by @XuQianJin-Stars in #5548
• docs: auto-build refactored namespace integrations doc by @jackye1995 in #5562

Performance Improvements 🚀

• perf: do not instrument self in multipart upload by @westonpace in #5416
• perf: various btree performance improvements by @westonpace in #5446
• perf: reuse session context by @wjones127 in #5462
• perf: offload IVF partition build to CPU pool by @BubbleCal in #5551
• perf: materialize the tokens after WAND done by @BubbleCal in #5572

Other Changes

• refactor: rename RowIdTreeMap to RowAddrTreeMap by @yanghua in #5266
• refactor: split dataset tests in a tests mod by @Xuanwo in #5387
• refactor: write bitmap index statistics in file instead by @Xuanwo in #5251
• refactor: consolidate logic between zonemap and bloomfilter indexes by @fenfeng9 in #5374
• refactor: use the same path for dedicated and packed blob by @Xuanwo in #5449
• refactor: expose take_blobs_by_addresses to python by @Xuanwo in #5474
• refactor: add store_prefix to lance-io's ObjectStore by @cmccabe in #5468
• refactor: rename RowIdMask to RowAddrMask by @yanghua in #5281
• refactor: support java 21, drop java 8 by @cmccabe in #5565

New Contributors

• @mackrorysd made their first contribution in #5315
• @valkum made their first contribution in #5372
• @rongou made their first contribution in #5439
• @camilesing made their first contribution in #5424
• @hfutatzhanghb made their first contribution in #5470
• @jonded94 made their first contribution in #5479
• @hushengquan made their first contribution in #5512
• @XuQianJin-Stars made their first contribution in #5548
• @YinZheng-Sun made their first contribution in #4839

Full Changelog: release-root/2.0.0-beta.N...v2.0.0-beta.5