v2.0.0-beta.9

What's Changed

Breaking Changes 🛠

• fix!: null handling when using NOT with scalar indices by @wjones127 in #5270
• feat!: track cumulative wall time in analyze plan by @wkalt in #5505
• fix!: check metric compatibility before using vector index by @wjones127 in #5609
• feat!: make v2 manifest default by @wojiaodoubao in #5656

New Features 🎉

• feat: support using FTS as a filter in vector search by @wojiaodoubao in #4928
• feat: support when_matched_delete in merge_insert by @jtuglu1 in #4939
• feat: add support for large minichunk size (u32) in format v2.2 by @niyue in #4959
• feat: support GEO RTree index by @ddupg in #5034
• feat: support global tag retrieval and improve tag api by @majin1102 in #5088
• feat: support create vector index distributedly by @chenghao-guo in #5117
• feat: support add sub-column to struct col by @wojiaodoubao in #5126
• feat: distributed range-based BTree index by @steFaiz in #5202
• feat: strategized plan compaction by @zhangyue19921010 in #5233
• feat: dataset supports deep_clone by @majin1102 in #5250
• feat: cleanup only scan managed files by @majin1102 in #5338
• feat: support map data type in lance format version 2.2 by @xloya in #5349
• feat: add RTree index spec in table format by @ddupg in #5360
• feat(java): support row lineage and cdf apis by @yanghua in #5362
• feat: disable default features on internal use by @valkum in #5372
• feat(cdf): support set start/end timestamp in cdf by @zhangyue19921010 in #5378
• feat(blob_v2): add external blob support by @Xuanwo in #5385
• feat(blob_v2): add dedicated blob support by @Xuanwo in #5406
• feat: fallback to CPU if GPU accelerating is unavailable by @BubbleCal in #5407
• feat(blob_v2): add packed blob support by @Xuanwo in #5413
• feat: allow python tracing / logging to be independently configured by @westonpace in #5415
• feat: add additional index APIs to support count rows split plan by @jackye1995 in #5447
• feat(java): support multi-bases for writing database by @ddupg in #5450
• feat(blob_v2): add BlobAray API for user input by @Xuanwo in #5451
• feat: upgrade lance-namespace to 0.3.1 and add missing apis by @jackye1995 in #5457
• feat(python): support cleanup_with_policy by @ddupg in #5458
• feat: support dropping sub-column of list(struct) by @wojiaodoubao in #5469
• feat(blob_v2): add GC support by @Xuanwo in #5473
• feat: add py.typed marker file by @jonded94 in #5479
• feat(python): expose the distance_range param in the Python scanner nearest config by @xloya in #5486
• feat(java): simplify the use of optional in jni by @ddupg in #5488
• feat(blob_v2): add Python API for Blob v2 by @Xuanwo in #5491
• feat(python): add DatasetBasePath stub to improve IDE hints by @ddupg in #5503
• feat(memtest): add macos support by @Xuanwo in #5510
• feat(java): add full text search api by @wojiaodoubao in #5563
• feat: support credentials vending in directory namespace by @jackye1995 in #5566
• feat: upgrade lance-namespace to 0.4.0 by @jackye1995 in #5568
• feat: add skip_merge for FTS index build by @BubbleCal in #5570
• feat(java): add builder-style scalar index params by @wojiaodoubao in #5581
• feat: optimize rle implementation by @Xuanwo in #5586
• feat: support FixedSizeList by @wkalt in #5593
• feat: add dictionary encoding for 64bit types like int64/double by @Xuanwo in #5594
• feat: support merge_insert with source dedupe on first seen value by @jackye1995 in #5603
• feat: support truncate table api by @zhangyue19921010 in #5604
• feat: add Error::External variant for preserving user errors by @wjones127 in #5606
• feat: upgrade lance-namespace to 0.4.5 by @jackye1995 in #5611
• feat: refactor use of Error::io by @lichuang in #5612
• feat(java): add detached flag to commitTransaction by @wojiaodoubao in #5626
• feat: add parts_searched metrics for FTS by @BubbleCal in #5627
• feat(oss): add sts token support for aliyun oss via storage_options by @hh23485 in #5632
• feat: merge-insert with primary key dedupe by @jackye1995 in #5633
• feat: allow configure temp dir size for datafusion exec by @jackye1995 in #5659
• feat(java): add support for optimizing indices by @majin1102 in #5663
• feat: make on arg optional for merge insert api by @yanghua in #5667
• feat: make OneShotPartitionStream pub by @timsaucer in #5672
• feat: add order to primary key by @touch-of-grey in #5683
• feat: use independent region manifest for MemWAL by @touch-of-grey in #5689
• feat: add stats() method to ObjectStoreRegistry by @wkalt in #5706
• feat: support dynamic context for lance namespace by @jackye1995 in #5710

Bug Fixes 🐛

• fix: correct null_count aggregation in boolean statistics collection by @YinZheng-Sun in #4839
• fix: remove logging for project_batch by @westonpace in #5267
• fix: stop documenting FTS index type, standardize on INVERTED by @mackrorysd in #5315
• fix: don't allow change blob version during update by @Xuanwo in #5386
• fix: take_blobs_by_indices fails with stable row IDs on fragment 1+ by @jmhsieh in #5392
• fix: respect index metric when user overrides by @BubbleCal in #5395
• fix: remove expensive clone in bitmap search by @westonpace in #5409
• fix: fix vector index prewarm index by @xloya in #5412
• fix: panic unwrap on None in decoder.rs by @camilesing in #5424
• fix: dir namespace cloud storage path removes one subdir level by @jackye1995 in #5464
• fix: make column name lookups case-insensitive by @wjones127 in #5465
• fix: ensure trailing slash is normalized in rest adapter by @jackye1995 in #5499
• fix(java): support FixedSizeList for java LanceField by @fangbo in #5509
• fix: head external manifest object happend 404 NotFound error by @hushengquan in #5512
• fix: json's arrow extension metadata missing by @Xuanwo in #5527
• fix: infer multivector sampling rows by @BubbleCal in #5534
• fix: support ManifestNamingSchemeV2 with unordered object stores by @wjones127 in #5539
• fix: merge_insert uses full schema path for reordered columns by @wjones127 in #5541
• fix: allow storage options provider without expires_at_millis by @jackye1995 in #5542
• fix(ci): use pull_request_target for fork PR reviews by @wjones127 in #5544
• fix: restore decrease max_fragment_id in manifest by @majin1102 in #5554
• fix: panic when lance.auto_cleanup.interval is set to 0 by @majin1102 in #5571
• fix(python): correct type hint for to_tensor_fn parameter by @AndreaBozzo in #5577
• fix: avoid panic while hitting non-null empty multi-vector by @Xuanwo in #5588
• fix: filter garbage entries from null maps during encoding by @wkalt in #5591
• fix: reduce verbosity of errors due to string conversion by @wjones127 in #5600
• fix: remove imports that are not needed by @westonpace in #5651
• fix: allow nearest applied in default_scan_options by @chenghao-guo in #5666
• fix: trait Array has been sealed in arrow new version by @Xuanwo in #5690
• fix: project_by_schema now reorders fields inside List types by @wjones127 in #5703
• fix: allocate too much memory for block max scores by @BubbleCal in #5718

Documentation 📚

• docs: fix Append call in distributed write guide by @rongou in #5439
• docs: fix and improve the description about row id by @yanghua in #5463
• docs: add specification for handling indices by @wjones127 in #5543
• docs: fix duplicate words in comments and error messages by @XuQianJin-Stars in #5548
• docs: add research paper link to the landing page by @prrao87 in #5549
• docs: auto-build refactored namespace integrations doc by @jackye1995 in #5562
• docs: rename RowIdTreeMap to RowAddrTreeMap in rtree.md by @ddupg in #5564
• docs: add docs for DuckDB extension by @prrao87 in #5578
• docs: update Lance-DuckDB docs to latest version 0.4.1 by @prrao87 in #5613

Performance Improvements 🚀

• perf: do not instrument self in multipart upload by @westonpace in #5416
• perf: various btree performance improvements by @westonpace in #5446
• perf: reuse session context by @wjones127 in #5462
• perf: offload IVF partition build to CPU pool by @BubbleCal in #5551
• perf: materialize the tokens after WAND done by @BubbleCal in #5572
• perf: compute HNSW level counts after build by @BubbleCal in #5590
• perf: improve SQ query speed by @BubbleCal in #5596
• perf: reuse zstd compressors in encoding by @wkalt in #5598
• perf: use binary search to skip documents by @BubbleCal in #5636
• perf: improve FTS indexing perf and reduce memory footprint by @BubbleCal in #5650
• perf: avoid copying tokens while merging by @BubbleCal in #5661
• perf: tighten WAND block score upper bound by @BubbleCal in #5668

Other Changes

• refactor: write bitmap index statistics in file instead by @Xuanwo in #5251
• refactor: rename RowIdTreeMap to RowAddrTreeMap by @yanghua in #5266
• refactor: rename RowIdMask to RowAddrMask by @yanghua in #5281
• refactor: consolidate logic between zonemap and bloomfilter indexes by @fenfeng9 in #5374
• refactor: split dataset tests in a tests mod by @Xuanwo in #5387
• refactor: use the same path for dedicated and packed blob by @Xuanwo in #5449
• refactor: add store_prefix to lance-io's ObjectStore by @cmccabe in #5468
• refactor: expose take_blobs_by_addresses to python by @Xuanwo in #5474
• refactor: support java 21, drop java 8 by @cmccabe in #5565
• refactor: allow switching to bitpack inside RLE by @Xuanwo in #5595

Full Changelog: release-root/2.0.0-beta.N...v2.0.0-beta.9