feat: add compound (multi-column) scalar index #5480
Conversation
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
It looks like your benchmark only compared to having a BTree index on one of the columns. Could you share results if you create a BTree index on both columns? Our query engine can combine the results of multiple index lookups, so I'd be curious how that compared to a compound index. |
|
Reran the benchmark, will push an update to it later but here's a look at the initial result:
|
wjones127
left a comment
wjones127
left a comment
There was a problem hiding this comment.
My immediate concerns on this PR aren't really about the compound index itself, but really about the changes to expression handling and other parts the generic index code. I think this would be the first index that covers multiple columns, and I think it needs careful design. I think we need a design discussion on that level before we are ready to move forward on the compound index itself.
I think reviewing both the API for multi-column indices and the concrete impl of the compound index might be too much. I think we should first outline the API changes needed to support multi-column indices (how are queries routed to them and run, for example).
I think the implementation itself looks pretty solid. There are lots of good tests, including property testing which I like to see.
|
I agree this probably needs to be broken up (11K LOC for one PR is a little daunting 😰). That being said it looks like a lot of well tested work. Thank you for starting this effort! I also agree with Will that a good start should be on supporting indexes on multiple columns both in the table format and the scanner. This is likely to be useful for other indexes as well. Maybe a good order can be...
At a glance it seems very reasonable and there is historical tradition for these kinds of compound indexes. However, like Will, I am also interested in understanding how compound indexes compare to multiple individual indexes. I do believe they can speed up certain classes of queries. Tenant + narrow range makes sense to me as the winner since the alternative requires crafting the entire bitmap of tenant which can be slow. Although in that case I might wonder if a bitmap index on tenant plus a btree index on the range column would perform similarly to this compound case. |
|
Appreciate the feedback, that makes sense. I figured this would need several iterations and probably redesigns, for the most part I wanted to get a POC done on our side to evaluate it vs other indexing strategies and it achieved that on our side. I guess my next question is what is the best approach for approaching the proper design and development stages for this? I'm interested to contribute to this along with any parallel/prerequisite ground work that you feel might be needed before multi column index can go in. |
Introduces compound B-tree scalar indices enabling efficient lookups on
multi-column predicates using a single index structure.
Features:
- CompoundSargableQuery and CompoundScalarQuery types for multi-column queries
- CompoundBTreeIndex with load(), search(), update(), and merge support
- CompoundQueryParser for extracting multi-column predicates from expressions
- Per-column page statistics for query pruning
- Support for prefix lookups, range queries, IN-lists, and IS NULL
- Fragment reuse remapping for compound indices during compaction
- Leftmost prefix rule semantics (2-8 columns)
API:
dataset.create_index(&["tenant_id", "timestamp"], IndexType::Scalar)
.with_index_name("tenant_time_idx")
.execute().await?;
Query patterns supported:
- Full key equality: col1 = X AND col2 = Y AND col3 = Z
- Prefix lookup: col1 = X or col1 = X AND col2 = Y
- Prefix + range: col1 = X AND col2 > Y
- IN-list: col1 IN (...) or col1 = X AND col2 IN (...)
- IS NULL: col1 = X AND col2 IS NULL
Adds a fourth benchmark scenario with two separate BTree indices (one on tenant_id, one on timestamp) to compare against the compound index. Key findings: - Compound index is 2-3x faster than single BTree for multi-column queries - Dual BTree can outperform compound for very narrow range queries - Compound wins for medium/wide ranges by avoiding intersection overhead
tomsanbear
force-pushed
the
feat/compound-index-clean
branch
from
411d221 to
4871f9c
Compare
3 participants
Summary
Introduces compound B-tree scalar indices to Lance, enabling efficient lookups on multi-column predicates using a single index structure instead of intersecting multiple single-column indices at query time.
Motivation
Many workloads filter on multiple columns simultaneously:
WHERE tenant_id = 'acme' AND timestamp > TWHERE status = 'active' AND created_at BETWEEN X AND YWHERE region = 'us-west' AND department = 'engineering'Compound indices store rows sorted by the combined key, following leftmost prefix semantics.
Query Patterns Supported
col1 = X AND col2 = Y AND col3 = Zcol1 = Xorcol1 = X AND col2 = Ycol1 = X AND col2 > Ycol1 IN (...)orcol1 = X AND col2 IN (...)col1 = X AND col2 IS NULLAPI
dataset .create_index(&["tenant_id", "timestamp"], IndexType::Scalar) .with_index_name("tenant_time_idx") .execute() .await?;Benchmark Results
Benchmarks run on multi-tenant time-series data with queries like
WHERE tenant_id = X AND timestamp > Y:Single fragment dataset:
Multi-fragment dataset (more realistic production scenario):
Production Experience
We've been running this implementation in our product with beta customers and are seeing stable, positive results. The index has been exercised against real multi-tenant time-series workloads.
A Note on the Implementation
I understand this is a large change. Many decisions, particularly around introducing
CompoundSargableQueryas a separate type and the associated changes for managing index creation, were made pragmatically to ease the initial implementation and reduce the effort of maintaining this fork until we could upstream it.Limitations