Skip to content

support rabitq in Hgraph#467

Merged
ShawnShawnYou merged 12 commits into
mainfrom
support-rabitq-in-Hgraph
Mar 14, 2025
Merged

support rabitq in Hgraph#467
ShawnShawnYou merged 12 commits into
mainfrom
support-rabitq-in-Hgraph

Conversation

@ShawnShawnYou

@ShawnShawnYou ShawnShawnYou commented Mar 3, 2025

Copy link
Copy Markdown
Collaborator
fashion-mnist-784-euclidean-bq-fp32:
    datapath: "/tbase-project/ann-benchmarks/data/fashion-mnist-784-euclidean.hdf5"
    type: "build,search"
    index_name: "hgraph"
    create_params: '{"dim":784,"dtype":"float32","metric_type":"l2","index_param":{
    "base_quantization_type":"rabitq",
    "use_reorder":true,
    "precise_quantization_type":"fp32",
    "max_degree":16,
    "ef_construction":200,
    "build_thread_count":16}}'
    search_params: '{"hgraph":{"ef_search":80}}'
    index_path: "/tmp/fashion-mnist-784-euclidean/index/hgraph_index_fp32"
    topk: 10
    search_mode: "knn"
    range: 0.5
    delete_index_after_search: false

gist-fp32:
    datapath: "/tbase-project/ann-benchmarks/data/gist-960-euclidean.hdf5"
    type: "build,search" # build, search
    index_name: "hgraph"
    create_params: '{"dim":960,"dtype":"float32","metric_type":"l2","index_param":{
    "base_quantization_type":"rabitq",
    "max_degree":32,
    "ef_construction":200,
    "build_thread_count":32,
    "use_reorder":true,
    "precise_quantization_type":"fp32",
    "precise_io_type":"block_memory_io"}}'
    search_params: '{"hgraph":{"ef_search":80}}'
    index_path: "/tmp/gist-960-euclidean/index/hgraph_index_fp32"
    topk: 10
    search_mode: "knn" # ["knn", "range", "knn_filter", "range_filter"]
    range: 0.5
    delete_index_after_search: false # free up storage space used by index

gist-fp32:
    datapath: "/tbase-project/ann-benchmarks/data/gist-960-euclidean.hdf5"
    type: "build,search" # build, search
    index_name: "hgraph"
    create_params: '{"dim":960,"dtype":"float32","metric_type":"l2","index_param":{
    "base_quantization_type":"rabitq",
    "max_degree":64,
    "ef_construction":300,
    "build_thread_count":32,
    "use_reorder":true,
    "precise_quantization_type":"fp32",
    "precise_io_type":"block_memory_io"}}'
    search_params: '{"hgraph":{"ef_search":80}}'
    index_path: "/tmp/gist-960-euclidean/index/hgraph_index_fp32"
    topk: 10
    search_mode: "knn" # ["knn", "range", "knn_filter", "range_filter"]
    range: 0.5
    delete_index_after_search: false # free up storage space used by index

results:
image
image
image


// TODO(ZXY): generate random orthogonal matrix
// validate rom
int retries = 3;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set as constant

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@wxyucs wxyucs added the version/0.14 1. Add IVF 2. Add Sparse 3. Async IO 1. 新增 IVF 2. 增加 Sparse 3. 异步 IO label Mar 5, 2025

@wxyucs wxyucs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@LHT129 LHT129 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShawnShawnYou
ShawnShawnYou force-pushed the support-rabitq-in-Hgraph branch 4 times, most recently from 11e8c68 to d18ba8b Compare March 10, 2025 15:29
@codecov

codecov Bot commented Mar 12, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 91.83673% with 8 lines in your changes missing coverage. Please review.

@@            Coverage Diff             @@
##             main     #467      +/-   ##
==========================================
- Coverage   91.97%   91.97%   -0.01%     
==========================================
  Files         173      173              
  Lines       10557    10615      +58     
==========================================
+ Hits         9710     9763      +53     
- Misses        847      852       +5     
Flag Coverage Δ
cpp 91.97% <91.83%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
common 95.21% <ø> (ø)
datacell 93.48% <88.88%> (-0.16%) ⬇️
index 91.04% <100.00%> (+0.03%) ⬆️
simd 87.27% <100.00%> (+0.05%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4b94cd...95e0045. Read the comment docs.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread src/algorithm/hgraph.cpp
search_param.ef = std::max(params.ef_search, k);
search_param.is_inner_id_allowed = ft;
search_param.topk = k;
search_param.topk = static_cast<int64_t>(search_param.ef);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix bug about unused rerank

@ShawnShawnYou
ShawnShawnYou force-pushed the support-rabitq-in-Hgraph branch from 33565be to 49909cf Compare March 13, 2025 08:24
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
@ShawnShawnYou
ShawnShawnYou force-pushed the support-rabitq-in-Hgraph branch from b572fbf to 95e0045 Compare March 14, 2025 02:25
@ShawnShawnYou
ShawnShawnYou merged commit c9d992e into main Mar 14, 2025
@ShawnShawnYou
ShawnShawnYou deleted the support-rabitq-in-Hgraph branch March 14, 2025 05:04
@wxyucs wxyucs linked an issue Mar 27, 2025 that may be closed by this pull request
LHT129 pushed a commit to LHT129/vsag that referenced this pull request Apr 16, 2026
support rabitq in hgraph

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
LHT129 pushed a commit that referenced this pull request May 11, 2026
support rabitq in hgraph

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: LHT129 <tianlan.lht@antgroup.com>
Sia-Sheerland pushed a commit to Sia-Sheerland/vsag that referenced this pull request Jun 26, 2026
support rabitq in hgraph

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: Sia Sheerland <x1075956441x@163.com>

Signed-off-by: Sia Sheerland <x1075956441x@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module/simd module/testing module/tools size/L version/0.14 1. Add IVF 2. Add Sparse 3. Async IO 1. 新增 IVF 2. 增加 Sparse 3. 异步 IO

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support binary quantization on hgraph

4 participants