Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: [restful v2]insert data with sparse vector failed #32214

Closed
1 task done
zhuwenxing opened this issue Apr 12, 2024 · 4 comments
Closed
1 task done

[Bug]: [restful v2]insert data with sparse vector failed #32214

zhuwenxing opened this issue Apr 12, 2024 · 4 comments
Assignees
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master/2.4
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2024-04-12 16:35:17 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.6.91:19530/v2/vectordb/collections/create, 
cost time: 0.18176007270812988, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': '9706deca-f8a7-11ee-8775-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_04_12_16_35_16_208097XQEaHVvS", "schema": {"autoId": true, "enableDynamicField": true, "fields": [{"fieldName": "book_id", "dataType": "Int64", "isPrimary": tr...se_float_vector", "indexName": "sparse_float_vector", "metricType": "IP", "indexConfig": {"index_type": "SPARSE_INVERTED_INDEX", "drop_ratio_build": "0.2"}}], "params": {"consistencyLevel": "Strong"}}, 
response: {"code":200,"data":{}} (milvus.py:17)
[2024-04-12 16:35:17 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.6.91:19530 (connectionpool.py:245)
[2024-04-12 16:35:17 - DEBUG - urllib3.connectionpool]: http://10.104.6.91:19530 "POST /v2/vectordb/collections/describe HTTP/1.1" 200 1062 (connectionpool.py:473)
[2024-04-12 16:35:17 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.6.91:19530/v2/vectordb/collections/describe, 
cost time: 0.031760215759277344, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': '9722f11e-f8a7-11ee-8775-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_04_12_16_35_16_208097XQEaHVvS"}, 
response: {"code":200,"data":{"aliases":[],"autoId":true,"collectionID":449023572199339053,"collectionName":"test_collection_2024_04_12_16_35_16_208097XQEaHVvS","consistencyLevel":"Strong","description":"","ena...":"SparseFloatVector"}],"indexes":[{"fieldName":"sparse_float_vector","indexName":"sparse_float_vector","metricType":"IP"}],"load":"LoadStateLoading","partitionsNum":64,"properties":[],"shardsNum":1}} (milvus.py:17)
[2024-04-12 16:35:17 - INFO - ci_test]: rsp: {'code': 200, 'data': {'aliases': [], 'autoId': True, 'collectionID': 449023572199339053, 'collectionName': 'test_collection_2024_04_12_16_35_16_208097XQEaHVvS', 'consistencyLevel': 'Strong', 'description': '', 'enableDynamicField': True, 'fields': [{'autoId': True, 'description': '', 'id': 100, 'name': 'book_id', 'partitionKey': False, 'primaryKey': True, 'type': 'Int64'}, {'autoId': False, 'description': '', 'id': 101, 'name': 'user_id', 'partitionKey': True, 'primaryKey': False, 'type': 'Int64'}, {'autoId': False, 'description': '', 'id': 102, 'name': 'word_count', 'partitionKey': False, 'primaryKey': False, 'type': 'Int64'}, {'autoId': False, 'description': '', 'id': 103, 'name': 'book_describe', 'params': [{'key': 'max_length', 'value': '256'}], 'partitionKey': False, 'primaryKey': False, 'type': 'VarChar'}, {'autoId': False, 'description': '', 'id': 104, 'name': 'sparse_float_vector', 'partitionKey': False, 'primaryKey': False, 'type': 'SparseFloatVector'}], 'indexes': [{'fieldName': 'sparse_float_vector', 'indexName': 'sparse_float_vector', 'metricType': 'IP'}], 'load': 'LoadStateLoading', 'partitionsNum': 64, 'properties': [], 'shardsNum': 1}} (test_vector_operations.py:762)
[2024-04-12 16:35:17 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.6.91:19530 (connectionpool.py:245)
[2024-04-12 16:35:17 - DEBUG - urllib3.connectionpool]: http://10.104.6.91:19530 "POST /v2/vectordb/entities/insert HTTP/1.1" 200 149 (connectionpool.py:473)
[2024-04-12 16:35:17 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.6.91:19530/v2/vectordb/entities/insert, 
cost time: 0.32216596603393555, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'Accept-Type-Allow-Int64': 'true', 'RequestId': '9738395c-f8a7-11ee-8775-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_04_12_16_35_16_208097XQEaHVvS", "data": [{"user_id": 0, "word_count": 0, "book_describe": "book_0", "sparse_float_vector": {"71": 0.22011702152827062, "22": 0.640575575292874, "12": 0.5995073799673194, "18": 0.010467555358631864, "56": 0.4984083794636065, "53": 0.7598312765346229, "93": 0.7590377414973769, "86": 0.020576136069964224, "10": 0.8135100939470645, "72": 0.42982283698103885, "48": 0.652487745202717, "117": 0.6468842585001653, "6": 0.887812910658206, "113": 0.0408829959146394, "42": 0.3008355328394139, "123": 0.993137062559305, "127": 0.5673001579612457, "43": 0.2636764410984209, "90": 0.10261155716811843, "87": 0.5008877733698391, "104": 0.13712476000414187, "70": 0.7159913540545428, "81": 0.4079812046598721, "59": 0.4744926740283981, "60": 0.24657736935350605, "23": 0.5000786311199478, "47": 0.24540943814751903, "27": 0.32462379198254465, "85": 0.718218387307329}, "dynamic_field_0": 0}, {"user_id": 1, "word_count": 1, "book_describe"... "17": 0.5987910511355903, "37": 0.2014749377025109, "43": 0.7574750250139034, "49": 0.018593256898462474, "23": 0.07096386348708594, "41": 0.6576710468320577, "126": 0.6723631426958634, "24": 0.1488112948192415, "84": 0.25052373017180585, "117": 0.016272838176228155, "52": 0.5934417219144067, "81": 0.788890841222369}, "dynamic_field_2998": 2998}, {"user_id": 99, "word_count": 2999, "book_describe": "book_2999", "sparse_float_vector": {"7": 0.7042044842106784, "121": 0.2893029742259038, "113": 0.017438838571064652, "102": 0.9366358817003854, "106": 0.03834217035061016, "5": 0.15623927509772095, "103": 0.7317876642120834, "26": 0.5564103819541828, "89": 0.8458836723985828, "4": 0.4186291882348825, "34": 0.08399385722767028, "45": 0.35698839518756487, "108": 0.2202957103470654, "8": 0.27659097335657223, "43": 0.7170373740009489, "3": 0.9831153371340142, "25": 0.8245073331244291, "126": 0.8257851470270356, "107": 0.5027146402886749, "84": 0.8683003196688509}, "dynamic_field_2999": 2999}]}, 
response: {"code":1804,"message":"fail to deal the insert data, error: fieldName: sparse_float_vector: invalid parameter[expected=][actual=SparseFloatVector]"} (milvus.py:20)
FAILED
testcases/test_vector_operations.py:726 (TestSearchVector.test_search_vector_with_sparse_float_vector_datatype[128-3000-True-True-True-1])
1804 != 200

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 12, 2024
@zhuwenxing
Copy link
Contributor Author

/assign @PowderLi

@zhuwenxing
Copy link
Contributor Author

/unassign @PowderLi
/assign @zhengbuqian
PTAL

@sre-ci-robot sre-ci-robot assigned zhengbuqian and unassigned PowderLi Apr 12, 2024
@zhengbuqian
Copy link
Collaborator

we haven't added sparse vector support to the restful api yet.

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 13, 2024
@yanliang567 yanliang567 added this to the 2.4.0 milestone Apr 13, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.0, 2.4.1 Apr 18, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.1, 2.4.2 May 7, 2024
sre-ci-robot pushed a commit that referenced this issue May 26, 2024
issue: #29419
also re-enabled an e2e test using restful api, which is previously
disabled due to #32214.

In restful api, the accepted json formats of sparse float vector are:

* `{"indices": [1, 100, 1000], "values": [0.1, 0.2, 0.3]}`
* {"1": 0.1, "100": 0.2, "1000": 0.3}

for accepted indice and value range, see
https://milvus.io/docs/sparse_vector.md#FAQ

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
@zhengbuqian
Copy link
Collaborator

/unassign
/assign @zhuwenxing

pls continue the testing for sparse restful, thanks!

yellow-shine pushed a commit to yellow-shine/milvus that referenced this issue Jul 2, 2024
issue: milvus-io#29419
also re-enabled an e2e test using restful api, which is previously
disabled due to milvus-io#32214.

In restful api, the accepted json formats of sparse float vector are:

* `{"indices": [1, 100, 1000], "values": [0.1, 0.2, 0.3]}`
* {"1": 0.1, "100": 0.2, "1000": 0.3}

for accepted indice and value range, see
https://milvus.io/docs/sparse_vector.md#FAQ

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

No branches or pull requests

4 participants