add sparse quantization (no simd optimization) by inabao · Pull Request #517 · antgroup/vsag

inabao · 2025-03-20T12:03:33Z

related to:#324

codecov · 2025-03-21T10:36:23Z

Codecov Report

Attention: Patch coverage is 86.42534% with 30 lines in your changes missing coverage. Please review.

@@            Coverage Diff             @@
##             main     #517      +/-   ##
==========================================
- Coverage   90.94%   90.70%   -0.25%     
==========================================
  Files         189      194       +5     
  Lines       11562    11812     +250     
==========================================
+ Hits        10515    10714     +199     
- Misses       1047     1098      +51

Flag	Coverage Δ
cpp	`90.70% <86.42%> (-0.25%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
common	`93.26% <ø> (-1.40%)`	⬇️
datacell	`92.14% <86.42%> (-0.54%)`	⬇️
index	`89.41% <ø> (-0.02%)`	⬇️
simd	`87.24% <ø> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 360af1e...8dc3b35. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jiaweizone

PART-I

jiaweizone · 2025-04-12T16:12:23Z

src/data_cell/sparse_vector_datacell.h

+    uint32_t current_offset_{0};
+};
+
+template <typename QuantTmpl, typename IOTmpl>


move all implements to .inl file and in file end #include this .inl file

src/data_cell/sparse_vector_datacell.h

jiaweizone · 2025-04-12T16:24:42Z

src/data_cell/sparse_vector_datacell.h

+void
+SparseVectorDataCell<QuantTmpl, IOTmpl>::BatchInsertVector(const void* vectors,
+                                                           InnerIdType count,
+                                                           InnerIdType* idx) {


idx -> idx_vec ?

jiaweizone · 2025-04-12T16:27:10Z

src/data_cell/sparse_vector_datacell.h

+    if (idx == nullptr) {
+        idx = idx_ptr.data();
+        for (InnerIdType i = 0; i < count; ++i) {
+            idx[i] = total_count_ + i;


add test case

jiaweizone · 2025-04-12T16:35:43Z

src/data_cell/sparse_vector_datacell.h

+void
+SparseVectorDataCell<QuantTmpl, IOTmpl>::InsertVector(const void* vector, InnerIdType idx) {
+    if (idx == std::numeric_limits<InnerIdType>::max()) {
+        idx = total_count_;


Why change the sparse vector idx ?

I have removed this part

src/data_cell/sparse_vector_datacell.h

jiaweizone · 2025-04-12T16:46:35Z

src/data_cell/sparse_vector_datacell.h

+        idx = total_count_;
+        ++total_count_;
+    } else {
+        total_count_ = std::max(total_count_, idx + 1);


Q: total_count_ in here is means 'max_idx’ ？

If the input ID exceeds the current total_count, update the total_count.

jiaweizone · 2025-04-13T15:21:30Z

src/data_cell/sparse_vector_datacell.h

+template <typename QuantTmpl, typename IOTmpl>
+MetricType
+SparseVectorDataCell<QuantTmpl, IOTmpl>::GetMetricType() {
+    return this->quantizer_->Metric();


jiaweizone · 2025-04-13T15:31:24Z

src/parameter.h

-        CHECK_ARGUMENT(json.contains("type"), "params must have type");  // TODO(LHT): "type" rename
-        return json["type"];
+        CHECK_ARGUMENT(json.contains(QUANTIZATION_TYPE_KEY),
+                       "params must have type");  // TODO(LHT): "type" rename


here let TryToParseType to get QUANTIZATION_TYPE_KEY type ?

tests/fixtures/fixtures.cpp

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

jiaweizone · 2025-04-18T03:00:27Z

src/data_cell/flatten_interface.cpp

 #include "io/io_headers.h"
 #include "quantization/quantizer_headers.h"
+#include "quantization/sparse_quantization/sparse_quantizer.h"
+#include "sparse_vector_datacell.inl"


move #include "sparse_vector_datacell.inl to sparse_vector_datacell.h file bottom

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

src/data_cell/sparse_vector_datacell.h

src/data_cell/sparse_vector_datacell.inl

src/quantization/sparse_quantization/sparse_quantizer.h

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

LHT129

LGTM

wxyucs

lgtm

jiaweizone · 2025-04-19T13:51:21Z

src/data_cell/flatten_datacell.h


    void
-    BatchInsertVector(const float* vectors, InnerIdType count, InnerIdType* idx) override;
+    BatchInsertVector(const void* vectors, InnerIdType count, InnerIdType* idx) override;


idx -> idx_vec

jiaweizone · 2025-04-19T13:51:39Z

src/data_cell/flatten_datacell.h

 template <typename QuantTmpl, typename IOTmpl>
 void
-FlattenDataCell<QuantTmpl, IOTmpl>::BatchInsertVector(const float* vectors,
+FlattenDataCell<QuantTmpl, IOTmpl>::BatchInsertVector(const void* vectors,


idx -> idx_vec

jiaweizone · 2025-04-19T13:52:02Z

src/data_cell/flatten_interface.h


    virtual void
-    BatchInsertVector(const float* vectors, InnerIdType count, InnerIdType* idx = nullptr) = 0;
+    BatchInsertVector(const void* vectors, InnerIdType count, InnerIdType* idx = nullptr) = 0;


idx -> idx_vec

jiaweizone · 2025-04-19T13:55:06Z

src/data_cell/sparse_vector_datacell.inl

+    }
+    auto* codes = reinterpret_cast<uint8_t*>(allocator_->Allocate(code_size));
+    quantizer_->EncodeOne((const float*)vector, codes);
+    uint32_t now_current_offset = 0;


now_current_offset -> new_offset ?

jiaweizone · 2025-04-19T13:57:14Z

src/quantization/sparse_quantization/sparse_quantizer_parameter.h

+
+    void
+    FromJson(const JsonType& json) override {
+    }


no implement ?

yes, there is no parameter for sparse quantizer now

inabao added kind/feature New feature or request version/0.15 labels Mar 20, 2025

inabao self-assigned this Mar 20, 2025

inabao requested review from LHT129, ShawnShawnYou, jiaweizone and wxyucs as code owners March 20, 2025 12:03

pull-request-size bot added the size/XL label Mar 20, 2025

mergify bot added the module/testing label Mar 21, 2025

inabao force-pushed the support_sparse_in_hgraph branch from 0e68ad7 to 539c8c2 Compare March 21, 2025 07:01

inabao force-pushed the support_sparse_in_hgraph branch 3 times, most recently from 478961e to e3503b8 Compare March 31, 2025 14:29

inabao force-pushed the support_sparse_in_hgraph branch 2 times, most recently from 3b5d927 to 10f7fac Compare April 9, 2025 08:37

jiaweizone reviewed Apr 12, 2025

View reviewed changes

jiaweizone reviewed Apr 13, 2025

View reviewed changes

pull-request-size bot added size/XXL and removed size/XL labels Apr 17, 2025

inabao added 10 commits April 17, 2025 14:43

add sparse quantization

9c2e980

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

modify

b3e102f

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

modify

cc9bb60

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

modify

6538bb9

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

modify

937fb66

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

modify

e81f268

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

modify

bdca4b3

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

modify

f8e0182

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

modify

645d8fd

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

modify

01bea36

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

inabao force-pushed the support_sparse_in_hgraph branch from 718cbf5 to 01bea36 Compare April 17, 2025 06:49

modify

0bfdcb6

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

jiaweizone reviewed Apr 18, 2025

View reviewed changes

modify

6f4edb1

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

wxyucs reviewed Apr 18, 2025

View reviewed changes

modify

8dc3b35

Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>

LHT129 approved these changes Apr 18, 2025

View reviewed changes

wxyucs approved these changes Apr 18, 2025

View reviewed changes

inabao merged commit 1edaaf1 into main Apr 19, 2025
22 of 23 checks passed

inabao deleted the support_sparse_in_hgraph branch April 19, 2025 04:48

jiaweizone reviewed Apr 19, 2025

View reviewed changes

Conversation

inabao commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jiaweizone left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LHT129 left a comment

Choose a reason for hiding this comment

Uh oh!

wxyucs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

inabao commented Mar 20, 2025 •

edited

Loading

codecov bot commented Mar 21, 2025 •

edited

Loading