support sparse vector in HGraph#604
Conversation
inabao
added
kind/feature
| get_data(const DatasetPtr& dataset, uint32_t index = 0) const { | ||
| if (data_type_ == DataTypes::DATA_TYPE_FLOAT) { | ||
| return dataset->GetFloat32Vectors() + index * dim_; | ||
| } else if (data_type_ == DataTypes::DATA_TYPE_SPARSE) { |
There was a problem hiding this comment.
current not support int8 ?
There was a problem hiding this comment.
there is no need for int8 in sparse vector now
| PARAMETER_DTYPE, | ||
| DATATYPE_FLOAT32, | ||
| DATATYPE_INT8, | ||
| DATATYPE_SPARSE, |
There was a problem hiding this comment.
format string not change
| size_t data_size, | ||
| std::vector<SparseVector>& parsed_vectors) { | ||
| std::vector<SparseVector>& parsed_vectors, | ||
| int64_t& dim) { |
There was a problem hiding this comment.
the dim is output param ?
we can define an sparse_vector struct to hold sparse_vector data and dim information ?
There was a problem hiding this comment.
I have changed the dim to max_len to denote the maximum value of valid dim.
55542d4 to
843fbed
Compare
Codecov ReportAttention: Patch coverage is @@ Coverage Diff @@
## main #604 +/- ##
==========================================
+ Coverage 91.05% 91.30% +0.24%
==========================================
Files 180 182 +2
Lines 10658 10865 +207
==========================================
+ Hits 9705 9920 +215
+ Misses 953 945 -8
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
| command: make test_parallel | ||
| no_output_timeout: 50m | ||
| no_output_timeout: 60m | ||
|
|
There was a problem hiding this comment.
I think 50m is enough, I have speedup some disk-based test
| route_graphs_(common_param.allocator_.get()), | ||
| dim_(common_param.dim_), | ||
| data_type_(common_param.data_type_), | ||
| use_reorder_(hgraph_param->use_reorder), |
There was a problem hiding this comment.
dim and data_type should place on InnerIndexInterface
Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com>
Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com> Signed-off-by: LHT129 <tianlan.lht@antgroup.com>
Signed-off-by: jinjiabao.jjb <jinjiabao.jjb@antgroup.com> Signed-off-by: Sia Sheerland <x1075956441x@163.com> Signed-off-by: Sia Sheerland <x1075956441x@163.com>
5 participants
close: #659
