Synchronization milvus with another milvus ?

[OsmanFarukO]

Hi all,
i want to run 2 standalone milvus instance that can sync each other. For example when i delete entity from milvus instance1 that instance will delete from instance2 (update, insert, delete). Is there any way that i can do that without hard way ?
Thank you :)

[xiaofan-luan]

Hi all, i want to run 2 standalone milvus instance that can sync each other. For example when i delete entity from milvus instance1 that instance will delete from instance2 (update, insert, delete). Is there any way that i can do that without hard way ? Thank you :)

Hi Osman, we are planing to support replication between milvus 2.0 cluster in the next 2 releases~ so far what we can offer is a migration tool to migrate the entire database.

Anyone interested please join the discussion about how to support data sync between milvus clusters (2.0 <-> 1.0, Faiss and more)

[OsmanFarukO]

Hello again, i wrote go api for sync standalone milvus(1.1.0) working in same kubernetes cluster. It can work any other location milvus too. Still working on this project. If you wanna checkout and support there is a link, thank you.
https://github.com/OsmanFarukO/go-shards

[yhmo]

Amazing, looks like the smart tool can: service discovery, service register, receive http request and pass request to a specify server. Are these milvus standalone instances share same storage and meta service? Or they have difference storage and meta service?

[OsmanFarukO]

Thanks for quick response, i designed for different location standalone milvus instances but i think it can work storage shared milvus instances too. I will test that.

[yhmo]

You need to be careful, the milvus 1.x only allow one writable-node in a cluster. Instances with different storage is ok because these instances are independent. If these instances share one storage and meta service, you must set the server_config.yaml like this:
For writable node:

# writable node
cluster:
  enable: true
  role: rw

For read-only node:

# read-only node
cluster:
  enable: true
  role: ro

In the cluster there must be only one writable node, because multiple writable nodes can crack the data files.

[OsmanFarukO]

Yes i familiar with mishard cluster. This project can allow you run synchronously run multiple standalone milvus or multiple location mishard cluster with 1 endpoint. Doesnt need run with mishard cluster at this point but it can do that too. I added my workflow to repository you can check that. Thank you.

[chtang-wish]

@xiaofan-luan xiao, we could like to deploy the multiple milvus in different region, is there any method I can sync data between them.

[yhmo]

Currently milvus has no ability to sync data with strong consistency across clusters.
There is a tool to backup/restore data between clusters: https://github.com/zilliztech/milvus-backup

[lipaysamart]

Hi, I need to migrate from Milvus 2.x to another 2.x version because we are using an older version of K8s, and upgrading K8s in place is too risky. Therefore, we need to migrate the service to a new cluster. Is there a migration tool that can achieve incremental synchronization? Milvus is installed as a Standalone version via Helm.

[yhmo]

milvus-cdc is a change data capture tool to sync increment data of upstream Milvus to downstream Milvus.
https://github.com/zilliztech/milvus-cdc

[yhmo]

Note: milvus-cdc requires milvus version >= 2.3.2

Read the doc about upgrade in different versions:
https://milvus.io/docs/v2.1.x/upgrade.md#Upgrade-Milvus-standalone
https://milvus.io/docs/v2.2.x/upgrade_milvus_standalone-helm.md
https://milvus.io/docs/v2.3.x/upgrade_milvus_standalone-helm.md

[lipaysamart]

milvus-cdc is a change data capture tool to sync increment data of upstream Milvus to downstream Milvus. https://github.com/zilliztech/milvus-cdc

Thank you for your prompt response. I am currently reviewing the CDC documentation, and for parts not mentioned in the document it seems that when using CDC, the existing collections will not be synchronized after the task is started; only collections created after the task starts will be synchronized. For best practices, it seems I should use the milvus-backup tool for data backup/recovery before initiating the CDC task.

[xiaofan-luan]

I don't think cdc can support very old milvus cluster.
what is the version of this cluster? can we use backup?