Reproducing Benchmarks and questions around latency

[vishal-kvn]

Hello

I have been trying to reproduce benchmarks by following the READMEs in lab2_sift1b_100m.md and my results are comparable to performance_benchmark.md. For this test I have used the sift_100m dataset, nprobe=128.

Hardware

Instance type: EC2 p3.8xl
GPU type: Tesla V100
GPU memory: 16 GB
vCPUs: 64
Memory: 244 GB
Local Storage: 1TB

Server Config

cache:
    cache_size: 25
    insert_buffer_size: 25GB
    preload_collection: ['sift_100m']
gpu:
    enable: true
    cache_size: 12GB
    gpu_search_threshold: 1001

Based on the above results, I have a few questions -

• Is it possible to achieve look up times < 100ms?
• Would Mishards help achieve this latency?
• For a Mishards proof of concept, can multiple servers be run on the same machine with different port mapping and adequate memory and cpu resources assigned?
• In the benchmark scripts, there are references to use_blas_threshold? Is this flag still valid?
• Regarding gpu_search_threshold, if it is set to 100 and there are 10 concurrent PUT requests to the Milvus server, with each request having 10 query vectors, will the GPU be used in this case?

Thanks in advance

[yhmo]

• Would Mishards help achieve this latency?
  Mishards distributes segments to multiple nodes, these nodes work together for a search request. For example, there are 100 segments in a collection, and there are 5 nodes deployed. Mishards will distributes the 100 segment to 5 nodes according to hash ring. Let's say, 15 segments on No.1 node, 20 segments on No.2 node, 22 segments on No.3 node, 18 segments on No.4 node, 35 segments on No.5 node. When a search request come in, Mishard master node send the search request to each node, and get 5 results from 5 nodes, reduce the 5 results into one final result.
  So, with more nodes, mishards can get better search performance.

• For a Mishards proof of concept, can multiple servers be run on the same machine with different port mapping and adequate memory and cpu resources assigned?
  The mishards use docker or k8s to deploy multiple nodes, these nodes can run on same machine, docker and k8s can allocate resource for them. But since they share same machine, the performance would not be better than one standalone server.

[vishal-kvn]

Following up on the throughput results. For throughput and latency load testing, the following configurations have been used -

• Config 1: TCP ELB => [4 Mishard nodes (c5.xl)] => [1 RW Milvus node(c5.9xl) & 4 RO Milvus node(c5.9xl)] => EFS
• Config 2: TCP ELB => [1 RW Milvus node(c5.9xl) & 4 RO Milvus node(c5.9xl)] => EFS

Docker images

• Milvus: 1.1.0
• pymilvus: 1.1.0

**AWS node specs **

• c5.xl: [4 vCPUs; 8 GB mem]
• c5.9xl: [36 vCPUs; 72 GB mem]

Load testing setup

• Framework: Locust
• Mode: Distributed load testing
• number of users: 360
• hatch rate: 1
• test duration(secs): 100
• number of Locust workers: 30

Dataset parameters

• size: 373M float vectors
• dim: 256
• index type: IVF_PQ

Milvus parameters

• cache_size: 60GB
• index_file_size: 4096MB
• m: 32
• nlist: 8192
• index type: PQ
• number of segments: 82
• number of index files: 82
• Insertion time: 2.8 hours
• Index build time: 48 minutes

Comparison

Config	Throughput(QPS)	avg(ms)	min(ms)	max(ms)	median(ms)	test duration (seconds)
ELB + [RW & RO nodes] (config 2)	242	127	42	310	130	100
ELB + Mishards (config 1)	162	116	39	375	110	100
ELB + [RW & RO nodes] (config 2)	318	353	34	16251	220	3600
ELB + Mishards (config 1)	164	653	33	7776	450	3600

Questions

• Are there any recommended best practices around instantiating multiple Milvus clients vs setting up a Milvus client per class.
• Based on the sizing-tool the memory size for an IVF_PQ index is 35GB. Therefore the index for all the segments fits in the memory of each EC2 instance. In this scenario is there any benefit to using Mishards given the overhead incurred in doing the reduce work(as you previously mentioned) and is also evident from the logs. If we end up not using Mishards, what would be the best way to support simultaneous reads and writes?
• In order to get better throughput, the log level has been set to ERROR. Are there any other flags that need to be set to get more throughput at the same or smaller latency?
• I read up on the Lucidwork's article where a QPS of 1500 was achieved, but the dataset size was less than 2M. Are there any references for a comparable dataset size?

Thanks in advance.

[yhmo]

• Are there any recommended best practices around instantiating multiple Milvus clients vs setting up a Milvus client per class.
  For pymilvus, it could throw connection error when using multiprocessing to send request by multiple Milvus clients. There is an issue about this: Multi-process insert encounter error #4863

• Based on the sizing-tool the memory size for an IVF_PQ index is 35GB. Therefore the index for all the segments fits in the memory of each EC2 instance. In this scenario is there any benefit to using Mishards given the overhead incurred in doing the reduce work(as you previously mentioned) and is also evident from the logs. If we end up not using Mishards, what would be the best way to support simultaneous reads and writes?

For example, I deploy Mishards with 1 write node and 2 ro nodes, insert 6M vectors(256 dim), do a query, the mishards node will print these logs:

2021-06-29 06:25:06,321 | INFO: Search TEST: topk=5 params={'nprobe': 12} (service_handler.py:330) (ThreadPoolExecutor-0_4)
2021-06-29 06:25:06,327 | INFO: Available servers: ['192.168.160.4:19530', '192.168.160.5:19530', '192.168.160.6:19530'] (file_based_hash_ring_router.py:118) (ThreadPoolExecutor-0_4)
2021-06-29 06:25:06,329 | INFO: Routing: {'192.168.160.5:19530': (['346', '348', '349', '350'], []), '192.168.160.6:19530': (['347'], []), '192.168.160.4:19530': (['351'], [])} (service_handler.py:135) (ThreadPoolExecutor-0_4)
2021-06-29 06:25:06,383 | INFO: Merge takes 4.3392181396484375e-05 (service_handler.py:101) (ThreadPoolExecutor-0_4)
2021-06-29 06:25:06,383 | INFO: SearchVector takes: 0.06209087371826172 (service_handler.py:372) (ThreadPoolExecutor-0_4)

From the log, we can know:

• There are 3 avaible servers(1 wr node + 2 ro nodes)
• There are 6 segments('346', '348', '349', '350', '347', '351'), distributed to 3 nodes, the first node got 4 segments, the other 2 nodes got one segment for each
• Reduce result cost 4.3392181396484375e-05 second
• The search request totally cost 0.06209087371826172 second, in this case we know the reduce work only takes 0.1% of time cost, it is not bottle neck

If we end up not using Mishards, user need to write a middle ware to do similar work to Mishards. The system must only has one write node, and multiple read nodes. This middle ware need to transfer create/insert/delete request to write node, send search request to read nodes and keep workload balance.

• In order to get better throughput, the log level has been set to ERROR. Are there any other flags that need to be set to get more throughput at the same or smaller latency?
  For 1.1.0, I think no other flags can get more throughput at the same or smaller latency. Huge dataset with high QPS is a big challenge, I did't see the query parameters in your experiment, reduce the "nprobe" and "topk" could improve performance, but this is a trade-off.
  Or increase read nodes may improve QPS, you can try 8 read nodes, I think 20GB mem for each node is ok.

The newest version 1.1.1 has the ability to keep data in GPU memory, this ability can get batter performance by using gpu devices.

• I read up on the Lucidwork's article where a QPS of 1500 was achieved, but the dataset size was less than 2M. Are there any references for a comparable dataset size?
  Not all users willing to share their statistics to public, some times we hear users share informations about their productiong on milvus in technicle meetings, but didn't got too much detail.

[vishal-kvn]

Thanks for the reply. A few follow up questions -

• Have you come across a use cases where using distributed ROs[ELB => 1 RW & 4ROs] has significantly better throughput as compared to a Mishard deployment[ELB => 4 Mishards => 1 RW & 4ROs]?
• Would using a different index type help, ie using IVF_SQ8 vs IVF_PQ? I will be testing it out soon.
• I increased the number of ROs from 4[ELB => 1 RW & 4ROs] to 8[ELB => 1 RW & 8ROs] and the throughput improved from 243 QPS to 303 QPS. This looks diminishing returns.

[yhmo]

Thanks for the reply. A few follow up questions -

• Have you come across a use cases where using distributed ROs[ELB => 1 RW & 4ROs] has significantly better throughput as compared to a Mishard deployment[ELB => 4 Mishards => 1 RW & 4ROs]?
• Would using a different index type help, ie using IVF_SQ8 vs IVF_PQ? I will be testing it out soon.
• I increased the number of ROs from 4[ELB => 1 RW & 4ROs] to 8[ELB => 1 RW & 8ROs] and the throughput improved from 243 QPS to 303 QPS. This looks diminishing returns.

• No, we didn't hear a use case compare distributed ROs vs Mishard deployment
• IVF_PQ is faster than IVF_FLAT/IVF_SQ8, but IVF_PQ's recall rate is not good
• Could you print out the logs of mishards for me, by command docker logs [mishards container id], like this:
  2021-06-29 06:25:06,329 | INFO: Routing: {'192.168.160.5:19530': (['346', '348', '349', '350'], []), '192.168.160.6:19530': (['347'], []), '192.168.160.4:19530': (['351'], [])} (service_handler.py:135) (ThreadPoolExecutor-0_4)

[vishal-kvn]

This is the routing from the Mishard docker logs -

Routing: {
'RO_IP_1:19530': (['38631', '38642', '38644', '38651', '38666', '38671', '38678'], []), # 7
'RO_IP_2:19530': (['38616', '38621', '38623', '38625', '38626', '38628', '38643', '38673', '38676', '38684'], []), # 10
'RO_IP_3:19530': (['38615', '38620', '38624', '38633', '38639', '38652', '38660', '38665', '38680'], []), # 9
'RO_IP_4:19530': (['38653', '38655', '38657', '38669', '38672', '38683'], []) # 6
'RO_IP_5:19530': (['38618', '38629', '38630', '38634', '38637', '38638', '38646', '38648', '38649', '38658', '38661', '38677'], []), # 12
'RO_IP_6:19530': (['38617', '38635', '38636', '38662', '38664', '38679', '38682'], []), # 7
'RO_IP_7:19530': (['38619', '38622', '38627', '38647'], []), # 4
'RO_IP_8:19530': (['38641', '38645', '38650', '38656', '38663', '38674', '38675'], []), # 7

'RW_IP_1:19530': (['38632', '38640', '38654', '38659', '38667', '38668', '38670', '38681'], []), # 8
}```
Total number of segments allocated is 70 which matches with the mysql db count.

Are there any performance numbers for `v2.0.0-rc1`. What are your recommendations for trying it?

[shiyu22]

Hi, you can refer to the performance_benchmark in Chinese, and we will update the English version later, and there is the answers about the questions:

• Is it possible to achieve look up times < 100ms?
  As the performance_benchmark, it can achieve if the nq = 1.
• Would Mishards help achieve this latency?
  If the number of queries is too large, then Milvus certainly can help.
• For a Mishards proof of concept, can multiple servers be run on the same machine with different port mapping and adequate memory and cpu resources assigned?
  Milvus itself is concurrently queried, so it is not recommended to create multiple Milvus on a single machine.
• In the benchmark scripts, there are references to use_blas_threshold? Is this flag still valid?
  The use_blas_threshold parameter is currently invalid, sorry that we will update the parameters.
• Regarding gpu_search_threshold, if it is set to 100 and there are 10 concurrent PUT requests to the Milvus server, with each request having 10 query vectors, will the GPU be used in this case?
  I think it will not use GPU, because the GPU will only be used if the number of single queries is greater than the gpu_search_threshold.

[vishal-kvn]

Thanks for pointing me to the latest benchmarks and answering my questions. I do have a few follow up questions:

• In the sift100_performance_benchmarks it says that the index creation took 229 seconds. When I did the same it took me 1000 secs or ~17 mins. Did you use the GPU for index creation?
• My throughput requirement is 670 rps, is this a good candidate for Mishards?
• In the Mishards example, a GPU docker image has been used, can a CPU docker image be used instead?
• Another thing that I noticed was that while building an index, there is no GPU utilization as per nvidia-smi is that expected? (I have enabled the gpu in the server_config.yml)

[shiyu22]

• In the sift100_performance_benchmarks it says that the index creation took 229 seconds. When I did the same it took me 1000 secs or ~17 mins. Did you use the GPU for index creation?
  Yes, we used GPU to build index.
• My throughput requirement is 670 rps, is this a good candidate for Mishards?
  Of course, you can try Mishards. In Lucidwork's case, they got 1500rps using distributed, and I think Mishards can meet your needs.
• In the Mishards example, a GPU docker image has been used, can a CPU docker image be used instead?
  We usually recommend using the GPU version to deploy write-only nodes and read nodes can use the CPU version because building indexes is faster with the GPU.
• Another thing that I noticed was that while building an index, there is no GPU utilization as per nvidia-smi is that expected? (I have enabled the gpu in the server_config.yml)
  There may be some problems with the settings, when building index with GPU nvidia-smi should show Milvus process.
  When setting up server_config.yml, you need not only set enable to true, but also configure build_index_devices, and also pay attention to use the GPU version of Milvus or restart Milvus to take effect.

[vishal-kvn]

Thank you for your reply. It's good to know that Mishards can scale to 1500 rps. I will be testing with Mishards next.

This is my server_config.yml -

gpu:
  enable: true
  cache_size: 12GB
  gpu_search_threshold: 1001
  search_devices:
    - gpu0
  build_index_devices:
    - gpu0

1. Does that look okay to you? Should I be changing any other config?

Also, can you please confirm your order of operations when building an index. I am following the steps from the benchmarks README.md ie:

• create collection
• build index
• load vectors

2. Should load vectors happen before build index?

3. Can you please answer this question from my previous response?
   In the Mishards example, it is recommended to use EFS so that the data is shared between multiple Milvus instances. I did try to use EFS instead of EBS for a stand alone Milvus instance and saw an order of magnitude degradation in the performance. Is there a recommended way to set up EFS instances?

Thanks in advance.

[yhmo]

Thank you for your reply. It's good to know that Mishards can scale to 1500 rps. I will be testing with Mishards next.

This is my server_config.yml -

gpu:
  enable: true
  cache_size: 12GB
  gpu_search_threshold: 1001
  search_devices:
    - gpu0
  build_index_devices:
    - gpu0

1. Does that look okay to you? Should I be changing any other config?

Also, can you please confirm your order of operations when building an index. I am following the steps from the benchmarks README.md ie:

• create collection
• build index
• load vectors

1. Should load vectors happen before build index?
2. Can you please answer this question from my previous response?
   In the Mishards example, it is recommended to use EFS so that the data is shared between multiple Milvus instances. I did try to use EFS instead of EBS for a stand alone Milvus instance and saw an order of magnitude degradation in the performance. Is there a recommended way to set up EFS instances?

Thanks in advance.

The config looks ok.

If your data is static, follow the steps:

1. create collection
2. insert vectors
3. create index

Else if your data is dynamic(continually streaming insert), follow the steps:

1. create collection
2. create index
3. insert vectors

EBS performance is better than EFS. In the documentation we mentioned EFS because EFS is easy to use for user.

[vishal-kvn]

Thank you for the response. For the benchmarking purpose I will be using a static dataset.