hgraph support extra info by lyxiong0 · Pull Request #486 · antgroup/vsag

lyxiong0 · 2025-03-07T02:46:03Z

Add new datacell typeExtraInfoDataCell
HGraph supports a new parameter extra_info_size
Modify HGraph related interface, including KnnSearch, Add etc.

related to issue #277

src/dataset_impl.h

inabao · 2025-03-11T15:11:23Z

src/algorithm/hgraph.cpp

        dists[j] = search_result.top().first;
        ids[j] = this->label_table_->GetLabelById(search_result.top().second);
+        if (extra_infos != nullptr) {
+            this->extra_infos_->GetExtraInfoById(search_result.top().second, tmp_extra_info.get());


inabao · 2025-03-11T15:11:44Z

src/algorithm/hgraph.cpp

        dists[j] = search_result.top().first;
        ids[j] = this->label_table_->GetLabelById(search_result.top().second);
+        if (extra_infos != nullptr) {
+            this->extra_infos_->GetExtraInfoById(search_result.top().second, tmp_extra_info.get());


why not pass the extra_infos directly?

inabao · 2025-03-11T15:18:18Z

src/algorithm/hgraph.cpp

        estimate_memory += block_memory_ceil(precise_memory, block_size);
    }

+    if (extra_info_size_ > 0 && this->extra_infos_->InMemory()) {


It is best to use whether this->extra_infos_ is nullptr as the condition to avoid core dumps.

inabao · 2025-03-11T15:20:16Z

src/data_cell/extra_info_datacell.cpp

+    this->io_->Deserialize(reader);
+}
+
+}  // namespace vsag


add new line

inabao · 2025-03-14T07:51:34Z

src/data_cell/extra_info_interface.h

+
+public:
+    InnerIdType total_count_{0};
+    InnerIdType max_capacity_{1000000};


why init the capacity as 1000000

Refer to the setting of FlattenDataCell, modify it to 800

inabao · 2025-03-14T07:55:46Z

tests/test_hgraph.cpp

+            vsag::Options::Instance().set_block_size_limit(origin_size);
+        }
+    }
+}


add new line

inabao · 2025-03-14T08:03:19Z

src/data_cell/extra_info_datacell.h

+    }
+
+public:
+    std::shared_ptr<BasicIO<MemoryBlockIO>> io_{nullptr};


Hardcoding the MemoryBlockIO in this part might not be the best design, as it would raise challenges for subsequent expansion of disk-based extroinfo. It is recommended to employ template, similar to how other data_cell implementations work, to resolve this issue.

As discussed earlier, Extra Info considers only the MemoryBlockIO type currently.
But we can reserve interfaces for other types by raising a std::invaliargument exception on initialization for types other than MemoryBlockIO.

codecov · 2025-03-17T14:29:34Z

Codecov Report

Attention: Patch coverage is 87.02703% with 24 lines in your changes missing coverage. Please review.

@@            Coverage Diff             @@
##             main     #486      +/-   ##
==========================================
- Coverage   90.66%   90.61%   -0.05%     
==========================================
  Files         177      182       +5     
  Lines       10846    11030     +184     
==========================================
+ Hits         9833     9995     +162     
- Misses       1013     1035      +22

Flag	Coverage Δ
cpp	`90.61% <87.02%> (-0.05%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
common	`95.28% <100.00%> (+0.03%)`	⬆️
datacell	`92.89% <83.68%> (-0.60%)`	⬇️
index	`88.63% <97.14%> (+0.07%)`	⬆️
simd	`87.27% <ø> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a227a2...69df18b. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wxyucs

lgtm

inabao

LGTM

LHT129

LGTM

Signed-off-by: lyxiong0 <bear507@163.com>

lyxiong0 requested review from LHT129, ShawnShawnYou, inabao, jiaweizone and wxyucs as code owners March 7, 2025 02:46

pull-request-size bot added the size/XL label Mar 7, 2025

mergify bot added the module/testing label Mar 7, 2025

lyxiong0 force-pushed the dev_extra_info branch from 37a9fec to 8682bbd Compare March 7, 2025 03:10

wxyucs reviewed Mar 7, 2025

View reviewed changes

src/dataset_impl.h Show resolved Hide resolved

inabao reviewed Mar 11, 2025

View reviewed changes

lyxiong0 force-pushed the dev_extra_info branch from 0a12a07 to c33253b Compare March 12, 2025 11:30

inabao added the version/0.14 label Mar 13, 2025

lyxiong0 force-pushed the dev_extra_info branch 2 times, most recently from a7a58f5 to fd4fc30 Compare March 13, 2025 13:31

inabao reviewed Mar 14, 2025

View reviewed changes

lyxiong0 force-pushed the dev_extra_info branch 2 times, most recently from 6501119 to 1be97eb Compare March 17, 2025 03:57

lyxiong0 force-pushed the dev_extra_info branch 2 times, most recently from 885ec7b to 08f7182 Compare March 19, 2025 06:33

pull-request-size bot added size/XXL and removed size/XL labels Mar 19, 2025

lyxiong0 force-pushed the dev_extra_info branch from eb50e07 to 35553db Compare March 20, 2025 03:22

wxyucs reviewed Mar 20, 2025

View reviewed changes

wxyucs approved these changes Mar 20, 2025

View reviewed changes

inabao approved these changes Mar 20, 2025

View reviewed changes

LHT129 approved these changes Mar 20, 2025

View reviewed changes

lyxiong0 force-pushed the dev_extra_info branch from 35553db to 4eb7b02 Compare March 20, 2025 07:13

implement extra info

59ace9a

Signed-off-by: lyxiong0 <bear507@163.com>

lyxiong0 force-pushed the dev_extra_info branch from ce46b4d to 60e4a73 Compare March 21, 2025 04:04

fix format

69df18b

Signed-off-by: lyxiong0 <bear507@163.com>

lyxiong0 force-pushed the dev_extra_info branch from 60e4a73 to 69df18b Compare March 21, 2025 06:53

wxyucs merged commit 015c091 into antgroup:main Mar 25, 2025
47 of 52 checks passed

wxyucs added the kind/feature New feature or request label Mar 26, 2025

wxyucs linked an issue Mar 26, 2025 that may be closed by this pull request

support extra info in index #277

Closed

Conversation

lyxiong0 commented Mar 7, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

inabao Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

wxyucs left a comment

Choose a reason for hiding this comment

Uh oh!

inabao left a comment

Choose a reason for hiding this comment

Uh oh!

LHT129 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

inabao Mar 14, 2025 •

edited

Loading

codecov bot commented Mar 17, 2025 •

edited

Loading