Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hybrid search #4226

Merged
merged 28 commits into from
Dec 14, 2023
Merged

Hybrid search #4226

merged 28 commits into from
Dec 14, 2023

Conversation

dureuill
Copy link
Contributor

@dureuill dureuill commented Nov 23, 2023 •
edited
Loading

Allows to perform hybrid search requests that combine the results of semantic and keyword search and automatically generate embeddings.

How to use

See feature description

Changes

  • work is based on WIP: Auto-generate embeddings for documents and queries #4213
  • milli::new search now takes an input universe directly, rather than computing it from a filter. This adds flexibility to require results on a subset of documents
  • vector search is now a regular ranking rule (akin to sort and geosort) and reports its score as a ScoreDetail
  • separate keyword search and vector search functions, vector search now respects (geo)sort ranking rules
  • add automatic embedding
  • add hybrid search

@dureuill dureuill mentioned this pull request Nov 27, 2023
Closed
45 tasks
@dureuill dureuill force-pushed the hybrid-search branch 2 times, most recently from dcd855b to cc4d5cc Compare November 29, 2023 08:54
@dureuill dureuill changed the base branch from autoembed to main December 5, 2023 17:33
dureuill
dureuill commented Dec 11, 2023
View reviewed changes
milli/Cargo.toml Show resolved Hide resolved
milli/Cargo.toml Show resolved Hide resolved
milli/Cargo.toml Outdated Show resolved Hide resolved
milli/src/vector/hf.rs Show resolved Hide resolved
@dureuill dureuill added this to the v1.6.0 milestone Dec 12, 2023
@dureuill
Add candle and hg_hub, updating a lot of deps in the process
21bcf32
@dureuill
Small commit to add hybrid search and autoembedding
13c2c6c
@dureuill
WIP arroy integration
dde3a04
@dureuill
WIP
cb4ebe1
@dureuill
WIP
fb539f6
@dureuill
prompt bifluor+
687d92f
@dureuill
Actually pass embedders on reindex
e56f160
@dureuill
Remove stuff, add distribution shift (WIP)
65e49b7
@dureuill
Fix geosort bug
11e2a2c
@dureuill
Fix same vector sort bug
d4715e0
@dureuill
Cosmetic change
abbe131
@dureuill
WIP multi embedders
922a640 
fixed template bugs
@dureuill
WIP
12940d7 
- manual embedder
- multi embedders OK
- clippy + tests OK
@dureuill
Various changes
e0cc775 
- DistributionShift in Search object (to be set from model in embed?)
- Fix issue where embedder index wasn't computed at search time
- Accept as default embedder either the "default" one, or the only embedder when there is only one
@dureuill
Update arroy
61bd2fb
@dureuill
Tests pass
806e5b6
@dureuill
Remove prompt strategy and fallback
b8e4709
@dureuill
Add settings routes
3c1a14f
@dureuill
Remove some settings
5b51cb0
@dureuill
Small adjustments to respect the spec
a4536b1
@dureuill
Copy link
Contributor Author

bors merge

@dureuill dureuill mentioned this pull request Dec 14, 2023
Closed
6 tasks
@meili-bors meili-bors
Copy link
Contributor

meili-bors bot commented Dec 14, 2023

Build succeeded:

@meili-bors meili-bors bot merged commit 2aede03 into main Dec 14, 2023
10 checks passed
@meili-bors meili-bors bot deleted the hybrid-search branch December 14, 2023 17:11
@meili-bot meili-bot added the v1.6.0 PRs/issues solved in v1.6.0 released on 2024-01-15 label Jan 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ManyTheFish ManyTheFish ManyTheFish approved these changes

Assignees
No one assigned
Labels
v1.6.0 PRs/issues solved in v1.6.0 released on 2024-01-15
Projects
None yet
Milestone
v1.6.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@dureuill @ManyTheFish @meili-bot