Can same indexed segments be searched by multiple nodes?

[seo-jinBro]

Hi milvus team. For better throughput, can same indexed segments be searched by multiple nodes?

We use milvus v1.1.1 on cluster mode with several read-only nodes.

Currently I'm doing stress test, I found that an indexed (or not) segment is searched by only one specific node.
(I did ssh connect to read-only nodes, and then tail logs...)

In my current setting, I think that milvus cannot distribute same indexed segments to multiple node. ( and duplicate same segment on a single node for better throughput )

Is that right? or am I missing something? If not, I hope this feature will be added in the next version.

Thank you.

[yhmo]

If you use Mishards, same segment cannot be searched by multiple nodes.
For example:

• A collection has 3 segments, we say s_1, s_2, s_3
• Mishrads deploys 3 read nodes, we say n_1, n_2, n_3
• Mishards will try send s_1 to n_1, s_2 to n_2, s_3 to n_3

The main purpose of Mishards is to handle huge dataset with lot of segments, each read node will take one part of each search request.
If the dataset is small, only has one or two segments, it will not be helpful with multiple read nodes.

[yhmo]

You also can create a service(by python or go, etc) to re-direct search request to multiple read nodes. Firstly deploy a write node and multiple read nodes by docker-composer or k8s, all clients connect to the service, the service forward insert/delete request to write node, search request to read node, one search request to one node.

[seo-jinBro]

Thanks for your replies.

I'm using Mishards on AWS, and made some microservices for serving over 100M datasets. (Backup, Indexing pipeline, divide two clusters for short-term and long-term, ... )

By doing theses things, I could make service which can handle lot of datasets, it cannot handle heavy traffics like 1k requests/second.

For static data, (== indexed segments) I hope that Mishards have an option to distribute same segments to multiple nodes for better throughput.

I hope to use Mishards for real-time services that are frequently called. (And I think many other Mishards users want it too.)

Could you consider adding a feature in the next version?

Thank you.

[yhmo]

We just complete the first RC release of Milvus v2.0, it is a real distribute system for vector/embedding search. We suppose the Mishards will be replaced by v2.0, so we only make minor fix for Mishards. In v1.1.0 and v1.1.1 you can see we fix a few bugs for Mishards, but we have no resource to make new features for it.

For search, Mishards mainly do two things:

1. tell each read node which segment it need to load and search, according to some kind of strategy
2. reduce search results from multiple read nodes, make a final result send to client

• Mishards use hash ring strategy to distribute segments, the logic code is in this file: /milvus/shards/mishards/router/plugins/file_based_hash_ring_router.py, the server_handler.py _do_query() method call to the _route() method. in _rote() method, firstly perform a SQL query to MySQL to get how many segments in the collection, secondly use hash ring to distribute segments to multiple nodes
• In the server_handler.py, after multiple nodes finish search, call _do_merge() method to make a final result.

For "distribute same segments to multiple nodes", need to rewrite a new pulgin like the file_based_hash_ring_router.py, and then rewrite the _do_merge() of server_handler.py.

I don't understand why distribute same segments to multiple nodes can get better throughput. Your dataset is 100M, it must contain lot of segments, assume there are 100 segments in the collection, and 10 read nodes, if each read node has 20 segments(some segments exist on multiple nodes), how can the results be merged into a final correct result?(result may contains duplicate items since some segment exist on multiple nodes)

[seo-jinBro]

Suppose desired requests/second is 10k, and dataset is 100M. No matter how segments is distributed finely, there's maximum requests/second for one single node like 1k request/second. (because scalability of cpu of one single cpu is limited)

For that reason, I think that same index segments should be distributed for better throughput.

I haven't seen python codes you said, (I will check that asap, thanks) for this purpose, would this structure be possible? Such that, mishards have map of <segments, list of nodes> and then load balance search requests to list of nodes and then merging it.

By the way, I'll check Milvus v2.0. Thanks for your replies.

[yhmo]

So assume a collection has 2 segments, and 10 ro nodes deployed, then each ro node will load the whole collection. All clients search requests send to a middle ware, this middle ware know which ro node is "busy" or "free", search request is send to "free" node.

If the solution is like this, the Mishards doesn't need to do result reduce work.
In the original code, the route() return a map [address, segment_list] to tell each read node which segment it need to load and search. If we take the new solution, the route() no need to return segment_list, just pick up a "free" node to do search and directly return result to client.