Skip to content

introduce IVF index implement#456

Merged
LHT129 merged 1 commit into
antgroup:mainfrom
LHT129:ivf_impl
Mar 10, 2025
Merged

introduce IVF index implement#456
LHT129 merged 1 commit into
antgroup:mainfrom
LHT129:ivf_impl

Conversation

@LHT129

@LHT129 LHT129 commented Feb 28, 2025

Copy link
Copy Markdown
Collaborator

issue: #276

@LHT129 LHT129 added kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 version/0.14 1. Add IVF 2. Add Sparse 3. Async IO 1. 新增 IVF 2. 增加 Sparse 3. 异步 IO labels Feb 28, 2025
@LHT129 LHT129 self-assigned this Feb 28, 2025
@LHT129
LHT129 force-pushed the ivf_impl branch 2 times, most recently from 2016613 to 0362266 Compare March 1, 2025 03:58
Comment thread src/algorithm/ivf.cpp Outdated
Comment thread src/algorithm/ivf.cpp Outdated
})";

ParamPtr
IVF::MappingExternalParamAndCheck(const JsonType& external_param,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MappingExternalParamAndCheck -> CheckAndMappingExternalParam ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes, we need check param valid after mapping

Comment thread src/algorithm/ivf_parameter.h Outdated
Comment thread src/algorithm/ivf_parameter.h Outdated
Comment thread src/algorithm/ivf_parameter.h Outdated

IVFSearchParameters obj;

// set obj.ef_search

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: ef_search for IVF is scan buckets count?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another param,not ef_search

Comment thread src/index/brute_force.cpp Outdated
@LHT129
LHT129 force-pushed the ivf_impl branch 3 times, most recently from 913700f to 39dc759 Compare March 3, 2025 12:57
@codecov

codecov Bot commented Mar 3, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 93.99142% with 14 lines in your changes missing coverage. Please review.

@@            Coverage Diff             @@
##             main     #456      +/-   ##
==========================================
+ Coverage   91.79%   91.97%   +0.18%     
==========================================
  Files         168      172       +4     
  Lines       10431    10495      +64     
==========================================
+ Hits         9575     9653      +78     
+ Misses        856      842      -14     
Flag Coverage Δ
cpp 91.97% <93.99%> (+0.18%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
common 95.02% <100.00%> (+0.26%) ⬆️
datacell 92.82% <100.00%> (+0.26%) ⬆️
index 91.20% <93.75%> (+0.15%) ⬆️
simd 87.95% <ø> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0e785e...b1c467f. Read the comment docs.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@LHT129
LHT129 force-pushed the ivf_impl branch 2 times, most recently from e4a7d1a to c36296b Compare March 4, 2025 02:45
@LHT129
LHT129 force-pushed the ivf_impl branch 7 times, most recently from f4a5a3e to a7fe5ba Compare March 5, 2025 07:38
Comment thread src/quantization/quantizer_parameter.cpp Outdated
Comment thread src/io/io_parameter.cpp Outdated
@LHT129
LHT129 force-pushed the ivf_impl branch 2 times, most recently from 70c3f70 to 7af865c Compare March 6, 2025 03:12
@LHT129
LHT129 force-pushed the ivf_impl branch 2 times, most recently from 0a2e331 to 743f069 Compare March 6, 2025 06:37
Signed-off-by: LHT129 <tianlan.lht@antgroup.com>

@wxyucs wxyucs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Comment thread src/algorithm/ivf.cpp
bucket_->ScanBucketById(dist.data(), computer, bucket_id);
for (int j = 0; j < bucket_size; ++j) {
if (filter == nullptr or filter->CheckValid(labels[j])) {
if (dist[j] <= radius + 2e-6 and dist[j] < cur_heap_top) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

convert to constant

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think all index need this constant, refactor in another PR

@inabao inabao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@LHT129
LHT129 merged commit 4093924 into antgroup:main Mar 10, 2025
@LHT129
LHT129 deleted the ivf_impl branch March 10, 2025 06:52
LHT129 added a commit to LHT129/vsag that referenced this pull request Apr 16, 2026
Signed-off-by: LHT129 <tianlan.lht@antgroup.com>
LHT129 added a commit that referenced this pull request May 11, 2026
Signed-off-by: LHT129 <tianlan.lht@antgroup.com>
Sia-Sheerland pushed a commit to Sia-Sheerland/vsag that referenced this pull request Jun 26, 2026
Signed-off-by: LHT129 <tianlan.lht@antgroup.com>
Signed-off-by: Sia Sheerland <x1075956441x@163.com>

Signed-off-by: Sia Sheerland <x1075956441x@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 module/testing size/XXL version/0.14 1. Add IVF 2. Add Sparse 3. Async IO 1. 新增 IVF 2. 增加 Sparse 3. 异步 IO

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants