Questions for not having a performance as expected #5986
Unanswered
YeloDriver
asked this question in
Q&A and General discussion
Replies: 4 comments 43 replies
-
This item default value is "false", set it to "true", milvus can hold data in GPU memory.
|
Beta Was this translation helpful? Give feedback.
1 reply
-
I'll use your script to redo the test and send you the result later |
Beta Was this translation helpful? Give feedback.
-
|
To improve QPS, you also can use multi-threads in client to send search request. I run this script on my machine, with My machine environment: |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
I use your script, and the result is as follows: |
Beta Was this translation helpful? Give feedback.
-
|
I just use another hdd hard disk to run run the script, it cost 63 seconds to insert 4M vectors. Your insert time cost is abnormal. Your build index time is 48 seconds, less than mine 120 seconds, that means your GPU is powerful than my GPU. The search performance should be better than mine. But your result is: The script execute search 1000 times, it the time cost is 5.0 second, QPS should be 1000/5=200. Your config is like this? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the server and client are on the same machine. And sorry for this mistake: |
Beta Was this translation helpful? Give feedback.
-
|
Ok, let compare the log. On my machine, each insert action(10000 vectors) cost 4 ~ 5 milliseconds. (the metadata backend is SQLite) For each search request, the debug.log will print these lines: On my machine, each search action(nq=1, k=20) cost 4 ~ 8 milliseconds |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
This is my log. This is the first to search, it takes about 200ms. And the rest takes about 4-6ms, sometimes 8ms |
Beta Was this translation helpful? Give feedback.
-
|
From your log, the time gap between two insert actions is 30.836-28.655=2181 ms In my log, the time gap between two insert actions is 4.325-4.208=117ms: During the time gap between two insert actions, the server and client do these things:
Could you change the insert() function and try again? Just move the "gen_vec_list" our side of the cycle: The first search request need to copy data from CPU memory into GPU memory, so it cost more time, this is make sense. |
Beta Was this translation helpful? Give feedback.
-
|
I've done what you said, and the result in the log is all the same, so I'll paste the log I've got yesterday. For the search performance, thanks for the explaination. But as yo said, my GPU is powerful than yours from the time of building index. Thus normally my search performance should be better than yours, but the result is the opposite. Isn’t that weird? Do you have any ideas? |
Beta Was this translation helpful? Give feedback.
-
|
Your GPU device is? My GPU card is Since the dataset is randomly generated. Vectors distribution also can affect search performance. Move the line "vectors = gen_vec_list(batch, seed=np.random.RandomState(i+13))" outside of the cycle will get lot of duplicated vectors, and IVF buckets become more intensive, search performance will decrease. For my test, move the line outside the cycle, search performance will decrease by about 40%. |
Beta Was this translation helpful? Give feedback.
-
|
My GPU card is |
Beta Was this translation helpful? Give feedback.
-
|
And for the poor insertion performance, you can try disable WAL in the server_config.yaml: |
Beta Was this translation helpful? Give feedback.
34 replies
-
|
Hello @yhmo, |
Beta Was this translation helpful? Give feedback.
-
Which tool can limit cpu speed? |
Beta Was this translation helpful? Give feedback.
-
|
@yhmo For Linux, I think we can use bios or |
Beta Was this translation helpful? Give feedback.
-
|
@yhmo Sorry to disturb you, but do you have time to do this test ？ |
Beta Was this translation helpful? Give feedback.
-
|
I tried cpupower to change cpu clock speed, but always get error: I think you can use multi-threads to do insert(), so that some decode/encode/transfer work can be executed parallelly. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello,
We are testing milvus to implement it into our project, as we're now using Faiss. But Milvus performs strangely according to my test result. The QPS is much slower than Faiss. For the proof of concept, I am using a dataset of 4 million vectors with 128dim. The QPS for nq = 1 in this situation is about 120, where as Faiss goes up to 644. The details of my concept is as follows.
Milvus 1.1.1 gpu
Hardware Config:
Milvus server config:
Insert and search config:
With this configuration, I got thes questions:
Beta Was this translation helpful? Give feedback.
All reactions