Data export to elasticsearch or json

[m-chael]

Hi all.
Need a way to import ~1mln records from milvus (v2.3.10) to Elasticsearch. Currently we have implemented a Java tool which iterates over the collection in batches and exports the data to json files. Then these jsons could be imported into elasticsearch. This seems to work on the test data, but we are a bit hesitant to run this on production. So we are looking for a more efficient and production-ready way of doing this.
The other alternative we were thinking of is to create the backup using milvus backup tool and then parse the backup files to restore the data and insert it into the elasticsearch instance, but given the complexity of the binlog file format that doesn't seem to be reasonable.

So, are there any better alternatives to that, or reading the data in batches is the only way?
Thanks, Mykhailo

[yhmo]

So far as I know, the query_iterator is the only way to export readable data from milvus.
The milvus-backup tool only copies the milvus binlog files to another directory, it doesn't parse the files. It is difficult to parse the binlog files by external applications.

[xiaofan-luan]

Hi all. Need a way to import ~1mln records from milvus (v2.3.10) to Elasticsearch. Currently we have implemented a Java tool which iterates over the collection in batches and exports the data to json files. Then these jsons could be imported into elasticsearch. This seems to work on the test data, but we are a bit hesitant to run this on production. So we are looking for a more efficient and production-ready way of doing this. The other alternative we were thinking of is to create the backup using milvus backup tool and then parse the backup files to restore the data and insert it into the elasticsearch instance, but given the complexity of the binlog file format that doesn't seem to be reasonable.

So, are there any better alternatives to that, or reading the data in batches is the only way? Thanks, Mykhailo

Hi @m-chael

1. we've build a tool named vts, https://github.com/zilliztech/vts and already support to export data from es and import to milvus and export data from milvus to milvus. Maybe you want to implement a logic to export milvus and write to ES?
2. if you always do it by batch, using milvus spark connector might be a good option
3. you can also check cdc https://github.com/zilliztech/milvus-cdc if you want to do incremental export to kafka. CDC server subscribe from milvus kafka and deliver to another kafka topic or another milvus

[m-chael]

@xiaofan-luan you are right, that's exactly what we want to accomplish - export milvus data and write to ES. Thanks for your suggestions, let me with a team explore them.