perf!: introduce compression and new indexing algo for FTS #3720

BubbleCal · 2025-04-22T05:00:53Z

This introduces:

compressing the posting lists and positions (4x smaller than before)
cache compressed posting lists in memory instead of plain posting lists
search over compressed posting lists, and decompress only needed blocks
new indexing algo, now it builds the index by partitioning by doc, and finally merge by size
the new index format has been changed so this is a breaking change
new search algo BlockMaxWAND

This also changes the default indexing params:

with_position -> false
stem -> true
remove_stop_words -> true
ascii_folding -> true

this also fix a bug in LanceIndexStore::copy(), which causes error if the src store is not the same as dest store (one on local disk but the other one on cloud)

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

…xing-perf

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

…xing-perf Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

…xing-perf Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Copilot

Pull Request Overview

This PR delivers major performance improvements for full‐text search by introducing a new compression algorithm for posting lists and a redesigned indexing strategy based on document partitioning and merging. It also updates the default indexing parameters, replaces the legacy TokenizerConfig with InvertedIndexParams, and fixes a bug in LanceIndexStore::copy() when source and destination stores differ.

Reviewed Changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
rust/lance/src/index.rs	Reordering and refinement of imports and test setup for inverted index configuration.
rust/lance/src/dataset.rs	Updated tests verifying index scanning and improved assertion messages.
rust/lance-index/src/scalar/inverted/wand.rs	Refactored posting iterator and WAND-related logic for better performance and clarity.
rust/lance-index/src/scalar/inverted/tokenizer.rs	Replaced TokenizerConfig with InvertedIndexParams and adjusted configuration methods.
python/src/dataset.rs, python/python/tests	Updated Python API usage and adjusted test expectations according to new defaults.
Other files	Refinements in compression, benchmark setup, and container API changes to support the new index format.

rust/lance-index/src/scalar/inverted/wand.rs

rust/lance-core/src/container/list.rs

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

…xing-perf

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

…xing-perf Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

…xing-perf Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

…date evaluation Signed-off-by: BubbleCal <bubble-cal@outlook.com>

westonpace

This is something of a YOLO. I've followed the basic changes I think. Overall it looks like a great addition. I appreciate the additional logging which will make it easier to debug. The compression looks good. Thanks for figuring this out. It looks like quite a bit of optimization, profiling, and debugging work!

westonpace · 2025-05-22T13:14:49Z

rust/lance-core/src/container/list.rs

-            len: 0,
-            limit,
-        }
+    pub fn with_capacity_limit(mut self, limit: u16) -> Self {


What happens if the user sets the capacity limit to something small after they've added items to the list?

oh i just realized this name confuses, it should be with_max_expand_cap, this effects only the new node cap

westonpace · 2025-05-22T13:24:29Z

rust/lance-index/src/scalar/inverted/builder.rs

-        self.write_tokens(dest_store).await?;
-        self.write_docs(dest_store).await?;
+    async fn write_metadata(&self, dest_store: &dyn IndexStore, partitions: &[u64]) -> Result<()> {
+        let metadata = HashMap::from_iter(vec![


Not for this PR but maybe in the future we could just allow the index store to read & write JSON files.

westonpace · 2025-05-22T13:48:37Z

rust/lance-index/src/scalar/inverted/encoding.rs

+
+/// decompress the posting list from a LargeBinaryArray
+/// returns a vector of (row_id, frequency) tuples
+#[allow(dead_code)]


Is this still dead code?

yeah, we decompress only needed block so it's never used, but leave it for convenient testing

westonpace · 2025-05-22T13:49:59Z

rust/lance-index/src/scalar/inverted/iter.rs

+    // skip to a doc id that is greater than or equal to the given doc id
+    // pub fn skip_to(&mut self, doc_id: u32) {
+    //     let
+    // }


westonpace · 2025-05-22T13:51:54Z

rust/lance-index/src/scalar/inverted/iter.rs

+}
+
+impl Iterator for TokenIterator<'_> {
+    type Item = (String, u32);


It's a little odd this is an iterator of String and not &'a str?

yes, it is, unfortunately the FST lib produces &[u8] and can't convert it to &str because of lifetime issue, we need str::from_raw_parts but it's still unstable

westonpace · 2025-05-22T13:54:55Z

rust/lance-index/src/scalar/lance_format.rs

        Self::schema(self)
    }
+
+    fn size(&self) -> u64 {


Where do you use this size?

I'm not entirely sure this is the exact size of the file (I think you would need to add a few more bytes)

it looks like I added them for some purpose but finally i'm not using it, remove it for now

westonpace · 2025-05-22T13:56:05Z

rust/lance-index/src/scalar/lance_format.rs

+                    .new_index_file(name, Arc::new(reader.schema().into()))
+                    .await?;
+
+                for offset in (0..reader.num_rows()).step_by(4096) {


We coudll maybe make this 4096 a parameter of the copy_index_file method, but that can be in the future when we hit a problem I suppose.

westonpace · 2025-05-22T14:06:17Z

Do we need to add a backwards compatibility test? To make sure that an old index is still readable by the new code? Also, to make sure we can read an old index, update the index, and then have a new index that still works?

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

… files for version 0.27.0 Signed-off-by: BubbleCal <bubble-cal@outlook.com>

BubbleCal · 2025-05-23T02:58:07Z

Do we need to add a backwards compatibility test? To make sure that an old index is still readable by the new code? Also, to make sure we can read an old index, update the index, and then have a new index that still works?

added

…ndling Signed-off-by: BubbleCal <bubble-cal@outlook.com>

BubbleCal added 16 commits April 11, 2025 16:53

new FTS indexing algo

b176f03

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

finish

6d244ec

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Merge branch 'main' of https://github.com/lancedb/lance into fts-inde…

ce61286

…xing-perf

improve

c1d04f9

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix clippy

45b2868

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

improve flushing

78dbfaa

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix clippy

e447bcf

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix clippy

8cb632e

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix

d1e1614

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

remove unused fields

f67cdef

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix type

8056e04

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix

cb2645a

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

remove unused import

9c9b3d3

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

remove unused Result

2d0c1f5

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

add exlinkedlist

401a221

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

add encoding

d795681

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

github-actions bot added python performance labels Apr 22, 2025

BubbleCal added 12 commits April 22, 2025 13:04

fix log

85f2162

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

implement delta encoding + bitpacking for posting list

1b4dd4f

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

add partition builder

bfce316

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Merge branch 'main' of https://github.com/lancedb/lance into fts-inde…

e55903c

…xing-perf Signed-off-by: BubbleCal <bubble-cal@outlook.com>

finish search part

bde9381

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

done indexing

7fe3167

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

add missed file

6136788

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix indexing

ca9f155

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix

488e373

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

finish compressing positions

a65c7b3

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Merge branch 'main' of https://github.com/lancedb/lance into fts-inde…

775c834

…xing-perf Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix accented test

bdcf375

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Copilot AI reviewed May 13, 2025

View reviewed changes

rust/lance-index/src/scalar/inverted/wand.rs Outdated Show resolved Hide resolved

rust/lance-core/src/container/list.rs Show resolved Hide resolved

BubbleCal marked this pull request as ready for review May 13, 2025 12:07

BubbleCal added 2 commits May 15, 2025 14:57

add block max wand

1458be6

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Merge branch 'main' of https://github.com/lancedb/lance into fts-inde…

2dc36c0

…xing-perf

BubbleCal mentioned this pull request May 16, 2025

Block max wand #3809

Closed

BubbleCal added 3 commits May 16, 2025 11:33

Merge branch 'main' of https://github.com/lancedb/lance into fts-inde…

b95165a

…xing-perf

Merge branch 'main' of https://github.com/lancedb/lance into fts-inde…

81dc9be

…xing-perf

store only doc weight for block scores

d8fb3fa

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

github-actions bot added the java label May 19, 2025

BubbleCal added 2 commits May 19, 2025 17:59

bump version

1af77f9

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

back

5235236

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

BubbleCal force-pushed the fts-indexing-perf branch from b7588a7 to 5235236 Compare May 20, 2025 05:06

BubbleCal added 7 commits May 20, 2025 13:41

Merge branch 'main' of https://github.com/lancedb/lance into fts-inde…

8ae100f

…xing-perf Signed-off-by: BubbleCal <bubble-cal@outlook.com>

remove debug logs

06a4ab7

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix

5994efd

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix

7ceee24

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

Merge branch 'main' of https://github.com/lancedb/lance into fts-inde…

60d1595

…xing-perf Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix: add remove_stop_words parameter to create_scalar_index in FTS test

eb545b0

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

fix: adjust doc_id increment in Wand implementation for correct candi…

1ca4334

…date evaluation Signed-off-by: BubbleCal <bubble-cal@outlook.com>

westonpace approved these changes May 22, 2025

View reviewed changes

BubbleCal added 2 commits May 23, 2025 10:22

fix: remove unimplemented size methods from IndexReader implementations

d579a31

Signed-off-by: BubbleCal <bubble-cal@outlook.com>

feat: add FTS backward compatibility test and generate necessary data…

042f148

… files for version 0.27.0 Signed-off-by: BubbleCal <bubble-cal@outlook.com>

feat: enhance FTS backward compatibility and optimize legacy index ha…

bad856f

…ndling Signed-off-by: BubbleCal <bubble-cal@outlook.com>

BubbleCal merged commit aa45c5d into lance-format:main May 26, 2025
26 of 28 checks passed

This was referenced May 27, 2025

perf: FTS efficiency optimizations #3713

Closed

perf: improve FTS indexing performance #3594

Closed

Enable remove_stop_words by default #3514

Closed

chenghao-guo mentioned this pull request Aug 21, 2025

feat: add create_fragment_inverted_index for distributed inverted index building #4514

Closed

perf!: introduce compression and new indexing algo for FTS #3720

perf!: introduce compression and new indexing algo for FTS #3720

Uh oh!

Conversation

BubbleCal commented Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westonpace commented May 22, 2025

Uh oh!

BubbleCal commented May 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BubbleCal commented Apr 22, 2025 •

edited

Loading