Skip to content

update several minor in conjugate graph#147

Merged
ShawnShawnYou merged 1 commit into
mainfrom
update-serveral-minor-in-conjugate-graph
Dec 2, 2024
Merged

update several minor in conjugate graph#147
ShawnShawnYou merged 1 commit into
mainfrom
update-serveral-minor-in-conjugate-graph

Conversation

@ShawnShawnYou

@ShawnShawnYou ShawnShawnYou commented Nov 17, 2024

Copy link
Copy Markdown
Collaborator

This PR contains several minor changes in conjugate graph:

  • support int8 in conjugate graph
  • support using conjugate graph in fresh_hnsw
  • add multi-thread test
  • modify search logic with conjugate graph
  • add maximum degree limit in conjugate graph

@ShawnShawnYou
ShawnShawnYou requested a review from inabao November 17, 2024 15:49
@ShawnShawnYou
ShawnShawnYou force-pushed the update-serveral-minor-in-conjugate-graph branch from 768d717 to 8a50245 Compare November 17, 2024 15:53
ShawnShawnYou
ShawnShawnYou commented Nov 18, 2024
View reviewed changes
Comment thread src/index/hnsw.cpp
// return result
while (results.size() > k) {
results.pop();
}

@ShawnShawnYou ShawnShawnYou Nov 18, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comparing with original search process, here only move partial pop operations from hnswalg to hnsw. So it wouldn't affect the search speed.

ShawnShawnYou
ShawnShawnYou commented Nov 18, 2024
View reviewed changes
Comment thread src/index/hnsw.cpp
Timer t(time_cost);
if (use_conjugate_graph_ and params.use_conjugate_graph_search) {
k = LOOK_AT_K;
}

@ShawnShawnYou ShawnShawnYou Nov 18, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here adjust k to provide enough information for conjugate graph to enhance result

ShawnShawnYou
ShawnShawnYou commented Nov 18, 2024
View reviewed changes
Comment thread src/impl/conjugate_graph.cpp
if (this->is_empty()) {
return 0;
}

@ShawnShawnYou ShawnShawnYou Nov 18, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here allow use conjugate graph as default setting without influencing search process

ShawnShawnYou
ShawnShawnYou commented Nov 18, 2024
View reviewed changes
Comment thread src/impl/conjugate_graph.cpp
auto& neighbor_set = conjugate_graph_[from_tag_id];
if (neighbor_set.size() >= MAXIMUM_DEGREE) {
return false;
}

@ShawnShawnYou ShawnShawnYou Nov 18, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here limit out degree to limit graph size and enhance latency

inabao
inabao reviewed Nov 18, 2024
View reviewed changes
Comment thread src/index/hnsw.cpp Outdated
try {
Timer t(time_cost);
if (use_conjugate_graph_ and params.use_conjugate_graph_search) {
k = LOOK_AT_K;

@inabao inabao Nov 18, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider the situation when k > LOOK_AT_K

Comment thread src/index/hnsw.cpp Outdated

auto vector = query->GetFloat32Vectors();
const void* vector;
if (type_ == DataTypes::DATA_TYPE_INT8) {

@inabao inabao Nov 18, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use get_vectors here to replace the judgement.

@ShawnShawnYou ShawnShawnYou Nov 19, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread src/index/hnsw.cpp
const int8_t* topk_data;
auto base = Dataset::Make();
auto generated_query = Dataset::Make();
if (type_ == DataTypes::DATA_TYPE_INT8) {

@inabao inabao Nov 18, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Comment thread src/index/hnsw.cpp Outdated
for (const int64_t& base_tag_id : base_tag_ids) {
try {
base->Float32Vectors(this->alg_hnsw_->getDataByLabel(base_tag_id));
if (type_ == DataTypes::DATA_TYPE_INT8) {

@inabao inabao Nov 18, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Comment thread src/index/hnsw.cpp
for (int d = 0; d < dim_; d++) {
generated_data.get()[d] = vsag::GENERATE_OMEGA * base->GetFloat32Vectors()[d] +
(1 - vsag::GENERATE_OMEGA) * topk_data[d];
if (type_ == DataTypes::DATA_TYPE_INT8) {

@inabao inabao Nov 18, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@ShawnShawnYou
ShawnShawnYou force-pushed the update-serveral-minor-in-conjugate-graph branch 2 times, most recently from 0c3eeb7 to 1068b44 Compare November 19, 2024 08:25
@ShawnShawnYou ShawnShawnYou changed the title Update several minor in conjugate graph update several minor in conjugate graph Nov 19, 2024
@ShawnShawnYou
ShawnShawnYou force-pushed the update-serveral-minor-in-conjugate-graph branch from 1068b44 to 2ea5ac9 Compare November 19, 2024 08:40
@ShawnShawnYou
ShawnShawnYou requested a review from LHT129 November 19, 2024 12:35
jiaweizone
jiaweizone reviewed Nov 19, 2024
View reviewed changes
Comment thread src/index/hnsw.cpp
->Owner(false)
->NumElements(num_element);
} else if (type_ == DataTypes::DATA_TYPE_INT8) {
base->Int8Vectors((int8_t*)vectors_ptr)->Dim(dim_)->Owner(false)->NumElements(num_element);

@jiaweizone jiaweizone Nov 19, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indent

Comment thread src/index/hnsw.cpp Outdated
} else if (type_ == DataTypes::DATA_TYPE_INT8) {
base->Int8Vectors((int8_t*)vectors_ptr)->Dim(dim_)->Owner(false)->NumElements(num_element);
} else {
throw std::invalid_argument(fmt::format("no support for this metric: {}", (int)type_));

@jiaweizone jiaweizone Nov 19, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no support for this data type: ?

jiaweizone
jiaweizone approved these changes Nov 20, 2024
View reviewed changes

@jiaweizone jiaweizone left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShawnShawnYou
ShawnShawnYou force-pushed the update-serveral-minor-in-conjugate-graph branch from 26f2be5 to 64e3f4d Compare November 25, 2024 08:07
wxyucs
wxyucs approved these changes Dec 2, 2024
View reviewed changes

@wxyucs wxyucs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@wxyucs wxyucs added the kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 label Dec 2, 2024
@ShawnShawnYou
ShawnShawnYou force-pushed the update-serveral-minor-in-conjugate-graph branch from ec373ec to dd48c34 Compare December 2, 2024 08:26
@pull-request-size pull-request-size Bot added size/XL and removed size/L labels Dec 2, 2024
@ShawnShawnYou
update
642866a 
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
@ShawnShawnYou
ShawnShawnYou force-pushed the update-serveral-minor-in-conjugate-graph branch from dd48c34 to 642866a Compare December 2, 2024 08:29
@pull-request-size pull-request-size Bot added size/L and removed size/XL labels Dec 2, 2024
@ShawnShawnYou
ShawnShawnYou merged commit 977b2c6 into main Dec 2, 2024
@ShawnShawnYou
ShawnShawnYou deleted the update-serveral-minor-in-conjugate-graph branch December 2, 2024 11:56
Roxanne0321 pushed a commit to Roxanne0321/vsag that referenced this pull request Mar 3, 2025
@ShawnShawnYou
update several minor in conjugate graph (antgroup#147)
d1f379e 
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
jiacai2050 pushed a commit to jiacai2050/vsag that referenced this pull request Mar 6, 2025
@ShawnShawnYou @jiacai2050
update several minor in conjugate graph (antgroup#147)
94f05be 
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
LHT129 pushed a commit that referenced this pull request May 11, 2026
@ShawnShawnYou @LHT129
update several minor in conjugate graph (#147)
6da36c0 
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: LHT129 <tianlan.lht@antgroup.com>
Sia-Sheerland pushed a commit to Sia-Sheerland/vsag that referenced this pull request Jun 26, 2026
@ShawnShawnYou
update several minor in conjugate graph (antgroup#147)
8560bd5 
Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>
Signed-off-by: Sia Sheerland <x1075956441x@163.com>

Signed-off-by: Sia Sheerland <x1075956441x@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@inabao inabao inabao left review comments
@jiaweizone jiaweizone jiaweizone approved these changes
@wxyucs wxyucs wxyucs approved these changes
@LHT129 LHT129 Awaiting requested review from LHT129 LHT129 is a code owner

Assignees

@ShawnShawnYou ShawnShawnYou

Labels

kind/feature Brand-new functionality or capabilities 引入全新的功能、新特性或新能力 size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ShawnShawnYou @jiaweizone @wxyucs @inabao