Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support more types of sparse vectors #293

Merged
merged 8 commits into from
Mar 29, 2024
Merged

Support more types of sparse vectors #293

merged 8 commits into from
Mar 29, 2024

Conversation

shanghaikid
Copy link
Contributor

Only support array, coo, csr, dict for nodejs.

array:

const sparseArray = [
          undefined,
          0.0,
          0.5,
          0.3,
          undefined,
          0.2]
      */

csr:

const sparseCSR = {
          indices: [2, 5, 8],
          values: [5, 3, 7]
      };

coo:

const sparseCOO = [
          { index: 2, value: 5 },
          { index: 5, value: 3 },
          { index: 8, value: 7 }
        ];

dict:

const sparseObject = {
          3: 1.5,
          6: 2.0,
          9: -3.5
      };

Signed-off-by: ryjiang <jiangruiyi@gmail.com>
Signed-off-by: ryjiang <jiangruiyi@gmail.com>
Signed-off-by: ryjiang <jiangruiyi@gmail.com>
Signed-off-by: ryjiang <jiangruiyi@gmail.com>
Signed-off-by: ryjiang <jiangruiyi@gmail.com>
Signed-off-by: ryjiang <jiangruiyi@gmail.com>
Signed-off-by: ryjiang <jiangruiyi@gmail.com>
@sre-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: shanghaikid

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: ryjiang <jiangruiyi@gmail.com>
Copy link

codecov bot commented Mar 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.15%. Comparing base (5034a82) to head (63decf1).

Additional details and impacted files
@@            Coverage Diff             @@
##              2.4     #293      +/-   ##
==========================================
+ Coverage   97.07%   97.15%   +0.07%     
==========================================
  Files          36       36              
  Lines        2120     2179      +59     
  Branches      567      585      +18     
==========================================
+ Hits         2058     2117      +59     
  Misses         54       54              
  Partials        8        8              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@shanghaikid shanghaikid merged commit 611a550 into 2.4 Mar 29, 2024
4 of 5 checks passed
@shanghaikid shanghaikid deleted the sparseTypes branch March 29, 2024 10:50
shanghaikid added a commit that referenced this pull request Apr 18, 2024
* init 2.4

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* sync 2.4 proto

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update CI

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add 2.4 index (#272)

* add INVERTED index

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add GPU_BRUTE_FORCE and GPU_CAGRA

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix index test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* support search group by (#273)

* support group by

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* remove  console

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update 2.4 version

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* support mmap (#275)

* support mmap

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* comment out test for now

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update proto

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* support search with multiple vectors (#278)

* update genCollectionParams to support multiple vectors field

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add test for generateInsertData about multiple vectors

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add compatible test for mutliple vectors

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update proto

Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>

* optimize search()

Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>

* part

Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>

* move buildSearchParams and formatSearchResult to utils

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* refactor

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* refactor part2

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* improve code
Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* refactor part3

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update part5

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* prefinish multivsearch

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add comments

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix comments

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix build

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>
Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>

* sync 2.4 proto

Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>

* Add sparse vector support (#285)

* add test data and const

Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>

* add sparse data generator

Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>

* fix create collection

Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>

* stash

Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>

* remove duplicate data sent to the sever

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* refactor data part1

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* finish insert

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add query

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* finish sparse vector

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix alter collection should run before load

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix http

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update test milvus verison

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add http test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>
Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix alterIndex

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* Support f16 & bf16  (#287)

* add f16 insert

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fp16 part2

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* stash

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* f16 part3

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* finish query

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* finish f16

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix type errors

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update parseFloat16VectorToBytes function

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add bf16 support

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update types

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* Support more types of sparse vectors (#293)

* generate different types of sparse vector

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix sparse array in js

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add sparse array test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add csr sparse vector test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add coo support

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* remove unused import

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* refine comments

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* refine comment

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* Sparse test update (#296)

* update sparse test case

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* log level

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* rename sparsevector test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* Add nq > 1 tests for sparse vectors and upgrade protos  (#297)

* add more test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add nq > 1 tests for sparse vectors

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update test version

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add nq>1 tests for hybridSearch (#298)

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* Fix single search failed on mutliple vectors collection if the anns field is specified. (#300)

* add more tests

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* Fix multiple test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update README.md

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add more tests for client contstructor (#301)

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* Add bfloat16 support (#302)

* add test for bf16

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add bf16 support

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* add transformers (#303)

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* rename types

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* update readme (#304)

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* Fix search transformers

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* Revert "Fix search transformers"

This reverts commit 2a14e0e.

* make transformer optional (#305)

* make transform optional

Signed-off-by: shanghaikid <jiangruiyi@gmail.com>

* update readme

Signed-off-by: shanghaikid <jiangruiyi@gmail.com>

* fix http test

Signed-off-by: shanghaikid <jiangruiyi@gmail.com>

* fix format test

Signed-off-by: shanghaikid <jiangruiyi@gmail.com>

* fix sparse array test

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: shanghaikid <jiangruiyi@gmail.com>
Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* feat: milvus api v2 (#295)

Signed-off-by: Shuyou <shuyou.zhan@zilliz.com>

* bump version

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

* fix build

Signed-off-by: ryjiang <jiangruiyi@gmail.com>

---------

Signed-off-by: ryjiang <jiangruiyi@gmail.com>
Signed-off-by: ruiyi.jiang <ruiyi.jiang@zilliz.com>
Signed-off-by: shanghaikid <jiangruiyi@gmail.com>
Signed-off-by: Shuyou <shuyou.zhan@zilliz.com>
Co-authored-by: Shuyoou <shuyou.zhan@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants