Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Search latency increase from 3s to 30s after upgrading from v2.2.2 to 2.2.0-20230213-4e26d626 #22163

Closed
1 task done
zhuwenxing opened this issue Feb 14, 2023 · 8 comments
Closed
1 task done

[Bug]: Search latency increase from 3s to 30s after upgrading from v2.2.2 to 2.2.0-20230213-4e26d626 #22163

zhuwenxing opened this issue Feb 14, 2023 · 8 comments
Assignees
@zhuwenxing
Labels
kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. resolution/fixed This issue or pull request is fixed severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.2.4

Comments

@zhuwenxing
Copy link
Contributor

zhuwenxing commented Feb 14, 2023 •
edited

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:2.2.0-20230213-4e26d626
- Deployment mode(standalone or cluster):standalone and cluster
- MQ type(rocksmq, pulsar or kafka):both    
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus==2.3.0.dev34
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

nq=10000
topk=100

data size: 100M

[2023-02-13T12:54:15.839Z] + python3 scripts/second_recall_test.py --host 10.101.167.165

[2023-02-13T12:54:17.720Z] 2023-02-13 12:54:17.540 | INFO     | __main__:search_test:33 - 

[2023-02-13T12:54:17.720Z] Search...

[2023-02-13T12:54:56.345Z] 2023-02-13 12:54:50.444 | INFO     | __main__:search_test:39 - search cost  32.9045 seconds

[2023-02-13T12:55:04.417Z] 2023-02-13 12:55:04.262 | INFO     | __main__:search_test:56 - recall=0.983

[2023-02-13T12:55:04.417Z] 2023-02-13 12:55:04.264 | INFO     | __main__:search_test:33 - 

[2023-02-13T12:55:04.417Z] Search...

[2023-02-13T12:55:43.040Z] 2023-02-13 12:55:37.508 | INFO     | __main__:search_test:39 - search cost  33.2442 seconds

[2023-02-13T12:55:51.105Z] 2023-02-13 12:55:50.872 | INFO     | __main__:search_test:56 - recall=0.983

[2023-02-13T12:55:51.105Z] 2023-02-13 12:55:50.873 | INFO     | __main__:search_test:33 - 

[2023-02-13T12:55:51.105Z] Search...

[2023-02-13T12:56:23.099Z] 2023-02-13 12:56:22.910 | INFO     | __main__:search_test:39 - search cost  32.0368 seconds

[2023-02-13T12:56:37.912Z] 2023-02-13 12:56:36.333 | INFO     | __main__:search_test:56 - recall=0.983

before upgrade:

[2023-02-13T12:35:34.306Z] Search...

[2023-02-13T12:35:34.306Z] search cost  3.8632 seconds

[2023-02-13T12:35:34.306Z] recall=0.983

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

collection name:sift_128_euclidean

image

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_kafka_for_release_cron/detail/deploy_test_kafka_for_release_cron/346/pipeline

log:
artifacts-kafka-cluster-upgrade-346-server-second-deployment-logs.tar.gz
artifacts-kafka-cluster-upgrade-346-server-first-deployment-logs.tar.gz
artifacts-kafka-cluster-upgrade-346-pytest-logs.tar.gz

Anything else?

It works well in 2.2.0-20230208-b83bbc04
passed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_for_release_cron/detail/deploy_test_for_release_cron/320/pipeline
@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 14, 2023
@zhuwenxing zhuwenxing added this to the 2.2.4 milestone Feb 14, 2023
@zhuwenxing zhuwenxing added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. labels Feb 14, 2023
@yanliang567
Copy link
Contributor

/assign @jaime0815
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 14, 2023
@xiaofan-luan
Copy link
Contributor

/assign @jiaoew1991
pls help on investigating

@zhuwenxing
Copy link
Contributor Author

It was not caused by upgrading since the search latency would be that large if you run this test case for a new Milvus cluster with the image of 2.2.0 branch eg. 2.2.0-20230216-fa8a9224.
So the origin latency of 2.2.0-20230216-fa8a9224 would be this scale. We can tell that from the reinstallation test in the job https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_for_release_cron/detail/deploy_test_for_release_cron/356/pipeline/126

[2023-02-16T11:12:30.919Z] + python3 scripts/first_recall_test.py --host 10.101.14.79

[2023-02-16T11:12:33.444Z] 2023-02-16 11:12:33.323 | INFO     | __main__:milvus_recall_test:43 - Create collection sift_128_euclidean

[2023-02-16T11:12:33.445Z] 2023-02-16 11:12:33.337 | INFO     | __main__:milvus_recall_test:50 - epoch: 0

[2023-02-16T11:12:35.961Z] 2023-02-16 11:12:35.504 | INFO     | __main__:milvus_recall_test:50 - epoch: 1

[2023-02-16T11:12:37.850Z] 2023-02-16 11:12:37.814 | INFO     | __main__:milvus_recall_test:50 - epoch: 2

[2023-02-16T11:12:39.738Z] 2023-02-16 11:12:39.532 | INFO     | __main__:milvus_recall_test:50 - epoch: 3

[2023-02-16T11:12:41.108Z] 2023-02-16 11:12:40.893 | INFO     | __main__:milvus_recall_test:50 - epoch: 4

[2023-02-16T11:12:42.476Z] 2023-02-16 11:12:42.268 | INFO     | __main__:milvus_recall_test:50 - epoch: 5

[2023-02-16T11:12:43.840Z] 2023-02-16 11:12:43.580 | INFO     | __main__:milvus_recall_test:50 - epoch: 6

[2023-02-16T11:12:45.207Z] 2023-02-16 11:12:44.975 | INFO     | __main__:milvus_recall_test:50 - epoch: 7

[2023-02-16T11:12:46.572Z] 2023-02-16 11:12:46.491 | INFO     | __main__:milvus_recall_test:50 - epoch: 8

[2023-02-16T11:12:47.936Z] 2023-02-16 11:12:47.829 | INFO     | __main__:milvus_recall_test:50 - epoch: 9

[2023-02-16T11:12:49.876Z] 2023-02-16 11:12:49.354 | INFO     | __main__:milvus_recall_test:50 - epoch: 10

[2023-02-16T11:12:50.806Z] 2023-02-16 11:12:50.631 | INFO     | __main__:milvus_recall_test:50 - epoch: 11

[2023-02-16T11:12:52.173Z] 2023-02-16 11:12:52.168 | INFO     | __main__:milvus_recall_test:50 - epoch: 12

[2023-02-16T11:12:53.563Z] 2023-02-16 11:12:53.533 | INFO     | __main__:milvus_recall_test:50 - epoch: 13

[2023-02-16T11:12:55.453Z] 2023-02-16 11:12:55.106 | INFO     | __main__:milvus_recall_test:50 - epoch: 14

[2023-02-16T11:12:56.819Z] 2023-02-16 11:12:56.504 | INFO     | __main__:milvus_recall_test:50 - epoch: 15

[2023-02-16T11:12:58.183Z] 2023-02-16 11:12:57.966 | INFO     | __main__:milvus_recall_test:50 - epoch: 16

[2023-02-16T11:12:59.548Z] 2023-02-16 11:12:59.366 | INFO     | __main__:milvus_recall_test:50 - epoch: 17

[2023-02-16T11:13:00.913Z] 2023-02-16 11:13:00.879 | INFO     | __main__:milvus_recall_test:50 - epoch: 18

[2023-02-16T11:13:02.800Z] 2023-02-16 11:13:02.316 | INFO     | __main__:milvus_recall_test:50 - epoch: 19

[2023-02-16T11:13:03.734Z] 2023-02-16 11:13:03.718 | INFO     | __main__:milvus_recall_test:63 - Insert 1000000 vectors cost 30.3814 seconds

[2023-02-16T11:13:03.734Z] 2023-02-16 11:13:03.719 | INFO     | __main__:milvus_recall_test:66 - Get collection entities...

[2023-02-16T11:13:07.899Z] 2023-02-16 11:13:07.342 | INFO     | __main__:milvus_recall_test:71 - 1000000

[2023-02-16T11:13:07.900Z] 2023-02-16 11:13:07.342 | INFO     | __main__:milvus_recall_test:73 - Get collection entities cost 3.6236 seconds

[2023-02-16T11:13:07.900Z] 2023-02-16 11:13:07.342 | INFO     | __main__:milvus_recall_test:78 - Create index...

[2023-02-16T11:13:34.390Z] 2023-02-16 11:13:31.458 | INFO     | __main__:milvus_recall_test:83 - Create index cost 24.1155 seconds

[2023-02-16T11:13:34.390Z] 2023-02-16 11:13:31.458 | INFO     | __main__:milvus_recall_test:87 - load collection...

[2023-02-16T11:13:39.633Z] 2023-02-16 11:13:38.603 | INFO     | __main__:milvus_recall_test:91 - load collection cost 7.1439 seconds

[2023-02-16T11:13:39.633Z] 2023-02-16 11:13:38.605 | INFO     | __main__:milvus_recall_test:94 - segments info: [segmentID: 439496447278925301

[2023-02-16T11:13:39.633Z] collectionID: 439496447278124381

[2023-02-16T11:13:39.633Z] partitionID: 439496447278124382

[2023-02-16T11:13:39.633Z] num_rows: 149914

[2023-02-16T11:13:39.633Z] state: Sealed

[2023-02-16T11:13:39.633Z] nodeIds: 1

[2023-02-16T11:13:39.634Z] , segmentID: 439496447278524868

[2023-02-16T11:13:39.634Z] collectionID: 439496447278124381

[2023-02-16T11:13:39.634Z] partitionID: 439496447278124382

[2023-02-16T11:13:39.634Z] num_rows: 175354

[2023-02-16T11:13:39.634Z] state: Sealed

[2023-02-16T11:13:39.634Z] nodeIds: 1

[2023-02-16T11:13:39.634Z] , segmentID: 439496447278124427

[2023-02-16T11:13:39.634Z] collectionID: 439496447278124381

[2023-02-16T11:13:39.634Z] partitionID: 439496447278124382

[2023-02-16T11:13:39.634Z] num_rows: 174374

[2023-02-16T11:13:39.634Z] state: Sealed

[2023-02-16T11:13:39.634Z] nodeIds: 1

[2023-02-16T11:13:39.634Z] , segmentID: 439496447278524866

[2023-02-16T11:13:39.634Z] collectionID: 439496447278124381

[2023-02-16T11:13:39.634Z] partitionID: 439496447278124382

[2023-02-16T11:13:39.634Z] num_rows: 174646

[2023-02-16T11:13:39.634Z] state: Sealed

[2023-02-16T11:13:39.634Z] nodeIds: 1

[2023-02-16T11:13:39.634Z] , segmentID: 439496447278925300

[2023-02-16T11:13:39.634Z] collectionID: 439496447278124381

[2023-02-16T11:13:39.634Z] partitionID: 439496447278124382

[2023-02-16T11:13:39.634Z] num_rows: 150086

[2023-02-16T11:13:39.634Z] state: Sealed

[2023-02-16T11:13:39.634Z] nodeIds: 1

[2023-02-16T11:13:39.634Z] , segmentID: 439496447278124426

[2023-02-16T11:13:39.634Z] collectionID: 439496447278124381

[2023-02-16T11:13:39.634Z] partitionID: 439496447278124382

[2023-02-16T11:13:39.634Z] num_rows: 175626

[2023-02-16T11:13:39.634Z] state: Sealed

[2023-02-16T11:13:39.634Z] nodeIds: 1

[2023-02-16T11:13:39.634Z] ]

[2023-02-16T11:13:39.634Z] 2023-02-16 11:13:38.606 | INFO     | __main__:milvus_recall_test:98 - wait for loading complete...

[2023-02-16T11:14:11.656Z] 2023-02-16 11:14:08.638 | INFO     | __main__:milvus_recall_test:101 - segments info: [segmentID: 439496447278524866

[2023-02-16T11:14:11.656Z] collectionID: 439496447278124381

[2023-02-16T11:14:11.656Z] partitionID: 439496447278124382

[2023-02-16T11:14:11.656Z] num_rows: 174646

[2023-02-16T11:14:11.656Z] state: Sealed

[2023-02-16T11:14:11.656Z] nodeIds: 1

[2023-02-16T11:14:11.656Z] , segmentID: 439496447278124426

[2023-02-16T11:14:11.656Z] collectionID: 439496447278124381

[2023-02-16T11:14:11.656Z] partitionID: 439496447278124382

[2023-02-16T11:14:11.656Z] num_rows: 175626

[2023-02-16T11:14:11.656Z] state: Sealed

[2023-02-16T11:14:11.656Z] nodeIds: 1

[2023-02-16T11:14:11.656Z] , segmentID: 439496447278124427

[2023-02-16T11:14:11.656Z] collectionID: 439496447278124381

[2023-02-16T11:14:11.656Z] partitionID: 439496447278124382

[2023-02-16T11:14:11.656Z] num_rows: 174374

[2023-02-16T11:14:11.656Z] state: Sealed

[2023-02-16T11:14:11.656Z] nodeIds: 1

[2023-02-16T11:14:11.656Z] , segmentID: 439496447278524868

[2023-02-16T11:14:11.656Z] collectionID: 439496447278124381

[2023-02-16T11:14:11.656Z] partitionID: 439496447278124382

[2023-02-16T11:14:11.656Z] num_rows: 175354

[2023-02-16T11:14:11.656Z] state: Sealed

[2023-02-16T11:14:11.656Z] nodeIds: 1

[2023-02-16T11:14:11.656Z] , segmentID: 439496447278925300

[2023-02-16T11:14:11.656Z] collectionID: 439496447278124381

[2023-02-16T11:14:11.656Z] partitionID: 439496447278124382

[2023-02-16T11:14:11.656Z] num_rows: 150086

[2023-02-16T11:14:11.656Z] state: Sealed

[2023-02-16T11:14:11.656Z] nodeIds: 1

[2023-02-16T11:14:11.656Z] , segmentID: 439496447278925301

[2023-02-16T11:14:11.656Z] collectionID: 439496447278124381

[2023-02-16T11:14:11.656Z] partitionID: 439496447278124382

[2023-02-16T11:14:11.656Z] num_rows: 149914

[2023-02-16T11:14:11.656Z] state: Sealed

[2023-02-16T11:14:11.656Z] nodeIds: 1

[2023-02-16T11:14:11.656Z] ]

[2023-02-16T11:14:11.656Z] 2023-02-16 11:14:08.638 | INFO     | __main__:milvus_recall_test:111 - Search...

[2023-02-16T11:14:33.540Z] 2023-02-16 11:14:32.762 | INFO     | __main__:milvus_recall_test:116 - search cost  24.1237 seconds

[2023-02-16T11:14:43.484Z] 2023-02-16 11:14:43.158 | INFO     | __main__:milvus_recall_test:127 - Calculate recall...

[2023-02-16T11:14:44.849Z] 2023-02-16 11:14:44.719 | INFO     | __main__:milvus_recall_test:134 - recall=0.983

[2023-02-16T11:14:44.849Z] 2023-02-16 11:14:44.719 | INFO     | __main__:milvus_recall_test:111 - Search...

[2023-02-16T11:15:52.466Z] 2023-02-16 11:15:41.953 | INFO     | __main__:milvus_recall_test:116 - search cost  57.2336 seconds

[2023-02-16T11:15:52.466Z] 2023-02-16 11:15:51.811 | INFO     | __main__:milvus_recall_test:127 - Calculate recall...

[2023-02-16T11:15:53.393Z] 2023-02-16 11:15:53.383 | INFO     | __main__:milvus_recall_test:134 - recall=0.983

[2023-02-16T11:15:53.393Z] 2023-02-16 11:15:53.383 | INFO     | __main__:milvus_recall_test:111 - Search...

[2023-02-16T11:17:01.017Z] 2023-02-16 11:16:51.723 | INFO     | __main__:milvus_recall_test:116 - search cost  58.3405 seconds

[2023-02-16T11:17:02.915Z] 2023-02-16 11:17:02.441 | INFO     | __main__:milvus_recall_test:127 - Calculate recall...

[2023-02-16T11:17:04.292Z] 2023-02-16 11:17:04.011 | INFO     | __main__:milvus_recall_test:134 - recall=0.983

[2023-02-16T11:17:04.292Z] 2023-02-16 11:17:04.054 | INFO     | __main__:milvus_recall_test:142 - {'int64': 2, 'float': 2.0}

[2023-02-16T11:17:04.292Z] 2023-02-16 11:17:04.054 | INFO     | __main__:milvus_recall_test:142 - {'int64': 4, 'float': 4.0}

[2023-02-16T11:17:04.292Z] 2023-02-16 11:17:04.055 | INFO     | __main__:milvus_recall_test:142 - {'int64': 6, 'float': 6.0}

[2023-02-16T11:17:04.292Z] 2023-02-16 11:17:04.055 | INFO     | __main__:milvus_recall_test:142 - {'int64': 8, 'float': 8.0}

@zhuwenxing
Copy link
Contributor Author

2.2.0-20230217-fbbbd6ea
job name: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_for_release_cron/detail/deploy_test_for_release_cron/363/pipeline

log with debug level:

artifacts-pulsar-standalone-reinstall-363-server-logs.tar.gz

artifacts-pulsar-standalone-reinstall-363-pytest-logs.tar.gz

@jiaoew1991
Copy link
Contributor

after downgrading knowhere to 1.3.7 it works fine.

https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_for_release_cron/detail/deploy_test_for_release_cron/382/pipeline/126

@liliu-z
Copy link
Member

liliu-z commented Feb 20, 2023 •
edited

/assign @liliu-z
/assign @hhy3
Suspect that this is caused by 1.3.8's change, which kills OMP in IVF-FLAT.
This will cause performance regression if any only if concurrency is 1 and with big NQ. As a tradeoff we get ~2x performance improvement on usual cases.

Need to confirm and check why 1.x's CI didn't alert us on it.

@jiaoew1991 jiaoew1991 removed their assignment Feb 20, 2023
@hhy3 hhy3 mentioned this issue Feb 23, 2023
Merged
@liliu-z liliu-z mentioned this issue Feb 23, 2023
Merged
@hhy3 hhy3 mentioned this issue Feb 23, 2023
Merged
@xiaofan-luan
Copy link
Contributor

/assign @zhuwenxing

@zhuwenxing
Copy link
Contributor Author

verified and passed with 2.2.0-20230310-b2ece6a5

[2023-03-13T07:41:07.880Z] + python3 scripts/first_recall_test.py --host 10.101.239.150

[2023-03-13T07:41:09.264Z] 2023-03-13 07:41:08.866 | INFO     | __main__:milvus_recall_test:73 - recall test for index type IVF_FLAT

[2023-03-13T07:41:11.163Z] 2023-03-13 07:41:10.830 | INFO     | __main__:milvus_recall_test:87 - Create collection sift_128_euclidean_IVF_FLAT

[2023-03-13T07:41:11.164Z] 2023-03-13 07:41:10.924 | INFO     | __main__:milvus_recall_test:94 - epoch: 0

[2023-03-13T07:41:14.451Z] 2023-03-13 07:41:14.424 | INFO     | __main__:milvus_recall_test:94 - epoch: 1

[2023-03-13T07:41:16.988Z] 2023-03-13 07:41:16.867 | INFO     | __main__:milvus_recall_test:94 - epoch: 2

[2023-03-13T07:41:19.527Z] 2023-03-13 07:41:19.064 | INFO     | __main__:milvus_recall_test:94 - epoch: 3

[2023-03-13T07:41:21.436Z] 2023-03-13 07:41:21.166 | INFO     | __main__:milvus_recall_test:94 - epoch: 4

[2023-03-13T07:41:23.342Z] 2023-03-13 07:41:23.249 | INFO     | __main__:milvus_recall_test:94 - epoch: 5

[2023-03-13T07:41:25.879Z] 2023-03-13 07:41:25.247 | INFO     | __main__:milvus_recall_test:94 - epoch: 6

[2023-03-13T07:41:27.789Z] 2023-03-13 07:41:27.542 | INFO     | __main__:milvus_recall_test:94 - epoch: 7

[2023-03-13T07:41:29.694Z] 2023-03-13 07:41:29.562 | INFO     | __main__:milvus_recall_test:94 - epoch: 8

[2023-03-13T07:41:32.231Z] 2023-03-13 07:41:31.748 | INFO     | __main__:milvus_recall_test:94 - epoch: 9

[2023-03-13T07:41:34.143Z] 2023-03-13 07:41:33.859 | INFO     | __main__:milvus_recall_test:94 - epoch: 10

[2023-03-13T07:41:36.056Z] 2023-03-13 07:41:36.006 | INFO     | __main__:milvus_recall_test:94 - epoch: 11

[2023-03-13T07:41:41.332Z] 2023-03-13 07:41:40.651 | INFO     | __main__:milvus_recall_test:94 - epoch: 12

[2023-03-13T07:41:43.301Z] 2023-03-13 07:41:42.900 | INFO     | __main__:milvus_recall_test:94 - epoch: 13

[2023-03-13T07:41:45.898Z] 2023-03-13 07:41:45.330 | INFO     | __main__:milvus_recall_test:94 - epoch: 14

[2023-03-13T07:41:47.807Z] 2023-03-13 07:41:47.438 | INFO     | __main__:milvus_recall_test:94 - epoch: 15

[2023-03-13T07:41:49.704Z] 2023-03-13 07:41:49.573 | INFO     | __main__:milvus_recall_test:94 - epoch: 16

[2023-03-13T07:41:52.248Z] 2023-03-13 07:41:51.677 | INFO     | __main__:milvus_recall_test:94 - epoch: 17

[2023-03-13T07:41:54.152Z] 2023-03-13 07:41:54.094 | INFO     | __main__:milvus_recall_test:94 - epoch: 18

[2023-03-13T07:41:56.056Z] 2023-03-13 07:41:55.975 | INFO     | __main__:milvus_recall_test:94 - epoch: 19

[2023-03-13T07:41:58.584Z] 2023-03-13 07:41:57.994 | INFO     | __main__:milvus_recall_test:107 - Insert 1000000 vectors cost 47.0701 seconds

[2023-03-13T07:41:58.584Z] 2023-03-13 07:41:57.995 | INFO     | __main__:milvus_recall_test:110 - Get collection entities...

[2023-03-13T07:42:01.115Z] 2023-03-13 07:42:00.531 | INFO     | __main__:milvus_recall_test:115 - 1000000

[2023-03-13T07:42:01.115Z] 2023-03-13 07:42:00.532 | INFO     | __main__:milvus_recall_test:117 - Get collection entities cost 2.5372 seconds

[2023-03-13T07:42:01.115Z] 2023-03-13 07:42:00.532 | INFO     | __main__:milvus_recall_test:121 - Create index...

[2023-03-13T07:42:07.678Z] 2023-03-13 07:42:06.832 | INFO     | __main__:milvus_recall_test:126 - Create index cost 6.2998 seconds

[2023-03-13T07:42:07.678Z] 2023-03-13 07:42:06.832 | INFO     | __main__:milvus_recall_test:130 - load collection...

[2023-03-13T07:42:15.827Z] 2023-03-13 07:42:14.442 | INFO     | __main__:milvus_recall_test:134 - load collection cost 7.6097 seconds

[2023-03-13T07:42:15.827Z] 2023-03-13 07:42:14.447 | INFO     | __main__:milvus_recall_test:137 - segments info: [segmentID: 440059334110554944

[2023-03-13T07:42:15.827Z] collectionID: 440059334110154259

[2023-03-13T07:42:15.827Z] partitionID: 440059334110154260

[2023-03-13T07:42:15.827Z] num_rows: 174646

[2023-03-13T07:42:15.827Z] state: Sealed

[2023-03-13T07:42:15.827Z] nodeIds: 1

[2023-03-13T07:42:15.827Z] , segmentID: 440059334110955560

[2023-03-13T07:42:15.827Z] collectionID: 440059334110154259

[2023-03-13T07:42:15.827Z] partitionID: 440059334110154260

[2023-03-13T07:42:15.827Z] num_rows: 150086

[2023-03-13T07:42:15.827Z] state: Sealed

[2023-03-13T07:42:15.827Z] nodeIds: 1

[2023-03-13T07:42:15.827Z] , segmentID: 440059334110154336

[2023-03-13T07:42:15.827Z] collectionID: 440059334110154259

[2023-03-13T07:42:15.827Z] partitionID: 440059334110154260

[2023-03-13T07:42:15.827Z] num_rows: 175626

[2023-03-13T07:42:15.827Z] state: Sealed

[2023-03-13T07:42:15.827Z] nodeIds: 3

[2023-03-13T07:42:15.827Z] , segmentID: 440059334110154334

[2023-03-13T07:42:15.827Z] collectionID: 440059334110154259

[2023-03-13T07:42:15.827Z] partitionID: 440059334110154260

[2023-03-13T07:42:15.827Z] num_rows: 174374

[2023-03-13T07:42:15.827Z] state: Sealed

[2023-03-13T07:42:15.827Z] nodeIds: 3

[2023-03-13T07:42:15.827Z] , segmentID: 440059334110554945

[2023-03-13T07:42:15.827Z] collectionID: 440059334110154259

[2023-03-13T07:42:15.827Z] partitionID: 440059334110154260

[2023-03-13T07:42:15.827Z] num_rows: 175354

[2023-03-13T07:42:15.827Z] state: Sealed

[2023-03-13T07:42:15.827Z] nodeIds: 8

[2023-03-13T07:42:15.827Z] , segmentID: 440059334110955559

[2023-03-13T07:42:15.827Z] collectionID: 440059334110154259

[2023-03-13T07:42:15.827Z] partitionID: 440059334110154260

[2023-03-13T07:42:15.827Z] num_rows: 149914

[2023-03-13T07:42:15.827Z] state: Sealed

[2023-03-13T07:42:15.827Z] nodeIds: 8

[2023-03-13T07:42:15.827Z] ]

[2023-03-13T07:42:15.827Z] 2023-03-13 07:42:14.450 | INFO     | __main__:milvus_recall_test:141 - wait for loading complete...

[2023-03-13T07:42:47.902Z] 2023-03-13 07:42:44.482 | INFO     | __main__:milvus_recall_test:144 - segments info: [segmentID: 440059334110554945

[2023-03-13T07:42:47.902Z] collectionID: 440059334110154259

[2023-03-13T07:42:47.902Z] partitionID: 440059334110154260

[2023-03-13T07:42:47.902Z] num_rows: 175354

[2023-03-13T07:42:47.902Z] state: Sealed

[2023-03-13T07:42:47.902Z] nodeIds: 8

[2023-03-13T07:42:47.902Z] , segmentID: 440059334110955559

[2023-03-13T07:42:47.902Z] collectionID: 440059334110154259

[2023-03-13T07:42:47.902Z] partitionID: 440059334110154260

[2023-03-13T07:42:47.902Z] num_rows: 149914

[2023-03-13T07:42:47.902Z] state: Sealed

[2023-03-13T07:42:47.902Z] nodeIds: 8

[2023-03-13T07:42:47.902Z] , segmentID: 440059334110554944

[2023-03-13T07:42:47.902Z] collectionID: 440059334110154259

[2023-03-13T07:42:47.902Z] partitionID: 440059334110154260

[2023-03-13T07:42:47.902Z] num_rows: 174646

[2023-03-13T07:42:47.902Z] state: Sealed

[2023-03-13T07:42:47.902Z] nodeIds: 1

[2023-03-13T07:42:47.902Z] , segmentID: 440059334110955560

[2023-03-13T07:42:47.902Z] collectionID: 440059334110154259

[2023-03-13T07:42:47.902Z] partitionID: 440059334110154260

[2023-03-13T07:42:47.902Z] num_rows: 150086

[2023-03-13T07:42:47.902Z] state: Sealed

[2023-03-13T07:42:47.902Z] nodeIds: 1

[2023-03-13T07:42:47.902Z] , segmentID: 440059334110154336

[2023-03-13T07:42:47.902Z] collectionID: 440059334110154259

[2023-03-13T07:42:47.902Z] partitionID: 440059334110154260

[2023-03-13T07:42:47.902Z] num_rows: 175626

[2023-03-13T07:42:47.902Z] state: Sealed

[2023-03-13T07:42:47.902Z] nodeIds: 3

[2023-03-13T07:42:47.902Z] , segmentID: 440059334110154334

[2023-03-13T07:42:47.902Z] collectionID: 440059334110154259

[2023-03-13T07:42:47.902Z] partitionID: 440059334110154260

[2023-03-13T07:42:47.902Z] num_rows: 174374

[2023-03-13T07:42:47.902Z] state: Sealed

[2023-03-13T07:42:47.902Z] nodeIds: 3

[2023-03-13T07:42:47.902Z] ]

[2023-03-13T07:42:47.902Z] 2023-03-13 07:42:44.482 | INFO     | __main__:milvus_recall_test:154 - Search...

[2023-03-13T07:42:50.444Z] 2023-03-13 07:42:49.881 | INFO     | __main__:milvus_recall_test:159 - search cost  5.3992 seconds

[2023-03-13T07:43:02.665Z] 2023-03-13 07:43:00.637 | INFO     | __main__:milvus_recall_test:170 - Calculate recall...

[2023-03-13T07:43:02.665Z] 2023-03-13 07:43:02.069 | INFO     | __main__:milvus_recall_test:177 - recall=0.968

[2023-03-13T07:43:02.665Z] 2023-03-13 07:43:02.069 | INFO     | __main__:milvus_recall_test:154 - Search...

[2023-03-13T07:43:09.330Z] 2023-03-13 07:43:08.661 | INFO     | __main__:milvus_recall_test:159 - search cost  6.5916 seconds

[2023-03-13T07:43:21.653Z] 2023-03-13 07:43:20.167 | INFO     | __main__:milvus_recall_test:170 - Calculate recall...

[2023-03-13T07:43:21.956Z] 2023-03-13 07:43:21.770 | INFO     | __main__:milvus_recall_test:177 - recall=0.968

[2023-03-13T07:43:21.956Z] 2023-03-13 07:43:21.770 | INFO     | __main__:milvus_recall_test:154 - Search...

[2023-03-13T07:43:30.481Z] 2023-03-13 07:43:29.764 | INFO     | __main__:milvus_recall_test:159 - search cost  7.9945 seconds

[2023-03-13T07:43:45.692Z] 2023-03-13 07:43:43.023 | INFO     | __main__:milvus_recall_test:170 - Calculate recall...

[2023-03-13T07:43:45.692Z] 2023-03-13 07:43:44.471 | INFO     | __main__:milvus_recall_test:177 - recall=0.968

[2023-03-13T07:43:45.692Z] 2023-03-13 07:43:44.522 | INFO     | __main__:milvus_recall_test:188 - {'int64': 2, 'float': 2.0}

[2023-03-13T07:43:45.692Z] 2023-03-13 07:43:44.523 | INFO     | __main__:milvus_recall_test:188 - {'int64': 4, 'float': 4.0}

[2023-03-13T07:43:45.692Z] 2023-03-13 07:43:44.523 | INFO     | __main__:milvus_recall_test:188 - {'int64': 6, 'float': 6.0}

[2023-03-13T07:43:45.692Z] 2023-03-13 07:43:44.523 | INFO     | __main__:milvus_recall_test:188 - {'int64': 8, 'float': 8.0}

@zhuwenxing zhuwenxing added the resolution/fixed This issue or pull request is fixed label Mar 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhuwenxing zhuwenxing

Labels
kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. resolution/fixed This issue or pull request is fixed severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.2.4
Development

No branches or pull requests
7 participants
@jiaoew1991 @jaime0815 @zhuwenxing @hhy3 @yanliang567 @xiaofan-luan @liliu-z