Skip to content

v9.0.0-rc.1

Pre-release
Pre-release

Choose a tag to compare

View all tags
@lance-community lance-community released this 15 Jul 19:27
v9.0.0-rc.1
cec0b7d

What's Changed

Breaking Changes 🛠

New Features 🎉

  • feat(namespace-dir): add alter table column operations (add/alter/drop) by @XuQianJin-Stars in #6273
  • feat: compile-time feature-gated Rust backtrace for JNI error diagnostics by @summaryzb in #6365
  • feat(lance-linalg): runtime SIMD dispatch for pre-Haswell x86_64 from-source builds by @tobocop2 in #6630
  • feat(namespace-dir): implement update_table and delete_from_table by @XuQianJin-Stars in #6923
  • feat(namespace-dir): implement alter_transaction for DirectoryNamespace by @XuQianJin-Stars in #6974
  • feat(mem_wal): support prefiltered LSM vector and FTS search by @touch-of-grey in #7138
  • feat(index): support distributed LabelList scalar index builds by @jackye1995 in #7223
  • feat(compaction): introduce RowAddrRemap structure to avoid remap OOM caused by HashMap by @zhangyue19921010 in #7237
  • feat: support cleanup explain for python and java by @yanghua in #7248
  • feat(schema_evolution): allow Dict <-> value-type casts in alter_columns by @valkum in #7289
  • feat: add SpillStore trait with local-disk implementation by @wjones127 in #7311
  • feat(blob-v2): configure packed blob file max size with field metadata by @jo-migo in #7322
  • feat(index): support Utf8View prefixes in SargableQueryParser by @wombatu-kun in #7351
  • feat: support COUNT(*) pushdown on stable row id datasets by @wkalt in #7360
  • feat(mem-wal): add ShardWriter::abort + Sealed manifest fence for drop-table by @hamersaw in #7361
  • feat(mem-wal): add ShardWriter::put_no_wait by @hamersaw in #7362
  • feat: add RLE v2 run length widths by @Xuanwo in #7376
  • feat: support hamming clustering by @brendanclement in #7379
  • feat(file): v2 writer/reader support columns of unequal length by @wjones127 in #7406
  • feat(mem_wal): row-level delete via tombstone rows by @hamersaw in #7417
  • feat(python): add list_with_delimiter, delete_file, and read_range to LanceFileSession by @jmhsieh in #7426
  • feat(python): expose fragment-reuse remap and delete-by-offset by @wkalt in #7438
  • feat(index): read a column's min/max from ZoneMap without a scan by @Ali2Arslan in #7463
  • feat: add ICU split tokenizer variant by @Xuanwo in #7474
  • feat(mem_wal): tombstone-preserving point lookup by @hamersaw in #7482
  • feat(mem_wal): non-blocking ShardWriter::delete_no_wait by @hamersaw in #7483
  • feat(bitpacking): add owned bitpacking codecs by @BubbleCal in #7496
  • feat(io): publish object store metrics via the metrics crate by @wjones127 in #7533
  • feat: data overlay files — model, feature flag, and write/commit path by @wjones127 in #7535
  • feat: resolve data overlay files on the take (and scan) read path by @wjones127 in #7536
  • feat(python): expose Lance metrics via OpenTelemetry by @wjones127 in #7537
  • feat: add blob write preparation API by @Xuanwo in #7558
  • feat: support update with blob encoding column by @nyl3532016 in #7579
  • feat(fts): impact skip data for posting lists by @BubbleCal in #7602
  • feat: support per-base storage options with base_. prefix by @jackye1995 in #7608
  • feat: support multi-base tables in merge insert with target base routing by @jackye1995 in #7610
  • feat(python): expose MemWAL shard delete by @u70b3 in #7649
  • feat: compress RLE child buffers by @Xuanwo in #7663
  • feat: support list blob scans by @Xuanwo in #7664
  • feat: support segment selection in pylance prewarm by @jackye1995 in #7677
  • feat: support nested field FTS by @Xuanwo in #7686
  • feat(java): expose MemWAL shard delete by @hamersaw in #7688
  • feat(java): forward table properties to declareTable on namespace create by @puchengy in #7711
  • feat(mem_wal): thread store_params + session through WAL write/read paths by @hamersaw in #7735
  • feat(python): add bulk packed blob writer API by @justinrmiller in #7743
  • feat(credentials): vend AWS credentials via AssumeRoleWithWebIdentity to avoid role chaining by @rpgreen in #7757
  • feat: support multi-segment indices in hamming clustering by @jackye1995 in #7758
  • feat: support wider hamming hashes by @jackye1995 in #7767

Bug Fixes 🐛

  • fix(python): fix tuple bug in _coerce_query_vector condition check by @hushengquan in #6555
  • fix: map object_store::Error::NotFound to Error::NotFound instead of Error::IO by @LuciferYang in #6569
  • fix: shift offsets after trimming values in merge_with_schema by @kaan-simbe in #6581
  • fix(transaction): reduce redundant code for new_version calculations by @yangshangqing95 in #6589
  • fix(index): keep HNSW IVF scratch dir alive during partition writing by @geserdugarov in #6980
  • fix: support manifests >5 GB via size-aware copy by @lixmgl in #7047
  • fix: route JSON index queries to the correct sub-parser by path by @ztorchan in #7072
  • fix(index): cap IVF train prefetch memory for nullable columns by @geserdugarov in #7143
  • fix: evaluate all list-element docs in FTS prefilter walk-the-allowlist branch by @Ar-maan05 in #7246
  • fix: apply per-segment filters and frag-reuse remap in BTree segment merge by @zhangyue19921010 in #7320
  • fix(index): drop stale scalar index entries after stable-row-id update by @wkalt in #7359
  • fix: reject DataReplacement racing concurrent Update/Delete/Merge by @wkalt in #7373
  • fix(ci): replace deprecated array.shape assignment for NumPy 2.5 by @zhangyue19921010 in #7384
  • fix(fts): enforce required terms for and queries by @BubbleCal in #7385
  • fix(index): use range block max for fts conjunction by @BubbleCal in #7387
  • fix(merge_insert): keep nested-field index correct on stable row id update by @jackye1995 in #7410
  • fix(update): keep nested-field index correct when updating a struct column by @jackye1995 in #7412
  • fix(io): clamp ObjectStore::io_parallelism() to at least 1 by @LuciferYang in #7414
  • fix(namespace): allow create_branch bootstrap on managed tables by @geruh in #7415
  • fix(fts): use async send in FTS index builder to prevent thread-pool … by @a-agmon in #7423
  • fix(rowids): union overlapping segments in RowAddrTreeMap::from by @wkalt in #7429
  • fix: relax DataReplacement conflict with concurrent Delete/Update by @brendanclement in #7433
  • fix(python): validate delete_rows offsets and add missing type stubs by @wkalt in #7448
  • fix(index): give each FTS partition a distinct scheduler base priority by @LuQQiu in #7449
  • fix(encoding): honor zstd compression scheme and level on large per-value buffers by @claydugo in #7460
  • fix(index): record IVF_HNSW index file sizes after writing footer by @Ali2Arslan in #7461
  • fix(core): allow all-null Map columns in schema evolution by @Ali2Arslan in #7462
  • fix(compaction): reject defer_index_remap with stable row IDs by @zhangyue19921010 in #7468
  • fix: convert Arrow JSON to Lance JSON in single-fragment create path by @xloya in #7469
  • fix: return Arrow JSON from FragmentSession.take by @xloya in #7470
  • fix: return Arrow JSON from FileFragment.take by @xloya in #7471
  • fix: convert Arrow JSON when updating columns via merge/update path by @xloya in #7472
  • fix: convert Arrow JSON in merge-insert full-fragment-rewrite fast path by @xloya in #7473
  • fix(index): preserve schema metadata when re-serializing loaded HNSW by @yanghua in #7476
  • fix(rowids): tolerate sparse overlapping chunks in the stable row id index by @wkalt in #7480
  • fix: account for SQ offset in dot distance by @Xuanwo in #7481
  • fix(merge-insert): apply Delete/Fail on the indexed-scan path by @hamersaw in #7484
  • fix(mem-wal): apply cross-generation block-list to in-memory scan arms by @hamersaw in #7489
  • fix: invalidate indices for fields rewritten by Merge by @wkalt in #7491
  • fix(index): implement with_io_priority for FailNewFileStore test store by @LuciferYang in #7495
  • fix(arrow): preserve inner nulls in convert_to_floating_point by @LuciferYang in #7498
  • fix: improve FM index query performance by @jackye1995 in #7507
  • fix: avoid stack overflow on deep logical filters by @Xuanwo in #7510
  • fix(compaction): exclude system indices from compaction binning by @xuanyu-z in #7516
  • fix(index): accept large_string (LargeUtf8) for BTREE and ZONEMAP scalar indices by @FANNG1 in #7525
  • fix(index): work around rustc nightly ICE in NGramIndexBuilder::stream_spill_reader by @westonpace in #7534
  • fix: recover from stale cached manifest size on read by @wkalt in #7542
  • fix(lance-encoding): rebase nested-list offsets correctly across pages by @yesunbmh in #7546
  • fix(mem-wal): fence writer on WAL persistence failure by @hamersaw in #7547
  • fix(mem_wal): use slice-aware size estimate for memtable flush threshold by @hamersaw in #7563
  • fix(index): translate address-domain scalar index results to row ids under stable row ids by @charleshuang119 in #7565
  • fix(lance-io): include goosefs feature in DEFAULT_CLOUD_BLOCK_SIZE cfg gate by @XuQianJin-Stars in #7570
  • fix: preserve structural page load errors by @ddupg in #7571
  • fix(cache): count key footprint in MokaCacheBackend eviction weight by @Ali2Arslan in #7573
  • fix(index): exclude shared metadata cache from LanceIndexStore deep size by @Ali2Arslan in #7574
  • fix(python): preserve PQ num_bits in model training by @ddupg in #7583
  • fix: respect PQ num_bits for numpy codebooks by @ddupg in #7586
  • fix(python): avoid CPU bf16 compile flake by @ddupg in #7587
  • fix: avoid scheduler deadlock on same-priority chunks by @Xuanwo in #7588
  • fix(fsst): correct decoder output-buffer size contract to 8x by @LuciferYang in #7589
  • fix: late-materialize blob columns read as binary by @wkalt in #7593
  • fix(fts): split tail-partition merge by the worker memory budget by @BubbleCal in #7601
  • fix: chunk ngram posting list writes by byte size to avoid i32 offset overflow by @lancedb-robot in #7607
  • fix: preserve DataFile base_id in DataReplacement commits by @jackye1995 in #7609
  • fix(goosefs): decouple OpenDAL root from URL path to avoid ObjectStore cache-key collision by @XuQianJin-Stars in #7612
  • fix: treat blob descriptors as opaque projections by @Xuanwo in #7618
  • fix(fts): use complete English stop-word list for ICU tokenizer by @LuQQiu in #7621
  • fix: relax test_query_delta_indices assertion for approximate index by @XuQianJin-Stars in #7622
  • fix(scanner): honor batch_readahead to bound v2 scan decode concurrency by @zhangyue19921010 in #7632
  • fix(fts): enforce fuzzy max_expansions globally across index partitions by @BubbleCal in #7634
  • fix(encoding): select RLE widths by encoded size by @Xuanwo in #7636
  • fix(lance-io): drop reservation when future is cancelled by @valkum in #7638
  • fix(deps): bump crossbeam-epoch to 0.9.20 for RUSTSEC-2026-0204 by @wjones127 in #7644
  • fix: return error instead of panicking on values too wide for miniblock by @zhangyue19921010 in #7650
  • fix: build list FTS indexes as row documents by @Xuanwo in #7656
  • fix(index): describe bitmap-less system indices in describe_indices by @hamersaw in #7667
  • fix: solve hang in train_streaming_coreset_ivf_model by @cmccabe in #7676
  • fix(index): batch IVF streaming partition search off the CPU pool by @a-agmon in #7680
  • fix(lance-core): avoid aliasing &mut Runtime in global_cpu_runtime by @LuciferYang in #7682
  • fix: avoid creating directory namespace manifest on read by @Xuanwo in #7687
  • fix(python): ignore PyTorch 3.14 script_method warning by @Xuanwo in #7689
  • fix: preserve case in generated field path expressions by @skyshineb in #7698
  • fix(encoding): tolerate legacy RLE chunks that overflow the declared value count by @jackye1995 in #7708
  • fix: round-trip empty blob values in the 2.1+ structural encoding by @wkalt in #7717
  • fix(encoding): reject stalled RLE miniblock encoding by @Ali2Arslan in #7729
  • fix(encoding): decode empty inline-bitpacked blocks by @Xuanwo in #7752
  • fix: reconstruct protobuf schema fields in linear time by @Xuanwo in #7766
  • fix: train IVF indexes on fragment subsets by @jackye1995 in #7768
  • perf: skip fragment-reuse index for compactions with no indexed data by @xuanyu-z in #7774
  • fix(dataset): reject all system column names on write by @LuciferYang in #7797
  • fix(ci): pin goosefs-sdk to 0.1.5 by @zhangyue19921010 in #7798

Documentation 📚

Performance Improvements 🚀

  • perf(knn): reduce memory for batch flat vector search by @LeoReeYang in #6950
  • perf(index): reduce TwoFileShuffler peak memory via interleave sort by @wjones127 in #7295
  • perf(index): add exact null-row bitmap to zone map and bloom filter by @westonpace in #7372
  • perf: speed up cold read up to 8x by lazily load column metadata by @Xuanwo in #7375
  • perf(fts): prune low-scoring conjunction candidates by @BubbleCal in #7386
  • perf: speed up ICU FTS index builds by 11% by @Xuanwo in #7393
  • perf(index): improve FTS search metadata caching by @Xuanwo in #7398
  • perf(fts): open fts segments as scalar indices by @BubbleCal in #7408
  • perf(fts): remove OR hot path bookkeeping by @BubbleCal in #7416
  • perf(index): parallelize FMIndex partition builds by @jackye1995 in #7422
  • perf(fts): use block max for or tail bounds by @BubbleCal in #7435
  • perf(fts): prewarm larger chunks concurrently by @BubbleCal in #7436
  • perf(index): coalesce concurrent scalar-index opens (single-flight) by @Ali2Arslan in #7464
  • perf: merge half-open range queries on the same BTree index by @xloya in #7477
  • perf(arrow): avoid per-element allocation in BFloat16Array::from by @LuciferYang in #7500
  • perf(index): skip FM-Index rebuild when merging a single fully-live segment by @jackye1995 in #7569
  • perf(dataset): reuse session-cached manifest when opening a dataset by @zhangyue19921010 in #7576
  • perf(filtered-read): lower hot-path read/decode spans to debug by @LuQQiu in #7590
  • perf(fts): cache the num_tokens-only DocSet on LazyDocSet by @BubbleCal in #7600
  • perf(fts): bulk MAXSCORE search path for top-k disjunctions by @BubbleCal in #7603
  • perf(arrow): zero-copy BFloat16Array::from(Vec) via Buffer::from_vec by @LuciferYang in #7614
  • perf(index): bulk conjunction path for FTS AND and phrase queries by @BubbleCal in #7624
  • perf(dataset): reuse session-cached manifest on checkout by @zhangyue19921010 in #7661
  • perf: increase default memory pool to 150MB with 40MB sort spill reservation by @westonpace in #7675
  • perf(index): pack prewarmed FTS posting groups by @BubbleCal in #7720
  • perf(filtered-read): consolidate take-shaped masked reads off the consumer by @LuQQiu in #7783
  • perf: support indexed metadata for structural projections by @Xuanwo in #7790

Other Changes

  • refactor(index): rely on total ordering for nan zonemap max by @HaochengLIU in #7049
  • refactor: remove as_vector_index from the Index trait by @westonpace in #7392
  • refactor(index): introduce RowIdRemapper trait to decouple ScalarIndexPlugin from FragReuseIndex by @westonpace in #7394
  • refactor(index): extract BasicTrainer trait from ScalarIndexPlugin by @westonpace in #7395
  • chore(deps): bump cmov from 0.5.3 to 0.5.4 in /test_data/fri_straddle_pre_6610/datagen by @app/dependabot in #7594
  • refactor(python): move tensorflow integration out of tree by @Xuanwo in #7685
  • refactor(index): introduce lance-index-core crate by @westonpace in #7713
  • refactor(java): rename WriteDatasetBuilder.tableProperties to properties by @puchengy in #7715
  • refactor(encoding): clarify mini-block repdef budget by @Xuanwo in #7751

Full Changelog: release-root/9.0.0-beta.N...v9.0.0-rc.1

Contributors

justinrmiller, valkum, and 47 other contributors
Assets 2
Loading