milvus-migration is a data migration tool for Milvus that supports importing Milvus 0.9.x ~ 1.x / faiss / elasticsearch7.x+ / Milvus2.3+ datas to milvus 2.x.
- Operating System Requirements
| Operating System | Supported Versions |
|---|---|
| CentOS | 7.5 or later |
| Ubuntu LTS | 18.04 or later |
- Software Requirements
| Software | Version |
|---|---|
| Milvus | 0.9.x, 1.x or 2.x |
| Elasticsearch | 7.x or 8.x |
| go | 1.20.2 or later |
- Data Format Support
| Source Data Type | Target Data Type |
|---|---|
| Milvus 0.9.x - 1.x | Milvus 2.x |
| Elasticsearch 7.x-8.x | Milvus 2.x |
| Faiss (Beta) | Milvus 2.x |
| Milvus 2.3 + | Milvus 2.x |
- Download the source code and build it:
git clone git@github.com:zilliztech/milvus-migration.git
go build- After building, you will obtain the
milvus-migrationbinary file. Place this file and themigration.yamlfile in the directory structure like below:
directory
---- milvus-migration
---- configs/migration.yaml
- Milvus1.x or Faiss to Milvus2.x migration
Run the command dump to dump the source data to numpy:
./milvus-migration dumpFinally, load the numpy files to Milvus 2.x successfully by using the load command:
./milvus-migration load- Elasticsearch to Milvus2.x migration:
Run the command start to migration es->2.x. here start == dump && load cmd
./milvus-migration start- Milvus 2.3+ to Milvus2.x migration:
./milvus-migration starthow to learn more about using migration tool, see examples doc below:
- es -> milvux2.x : migrate_es_doc.
- milvus1.x -> milvux2.x : migrate_1.x_doc.
- faiss -> milvux2.x ( Beta) : migrate_faiss_doc.
- milvus2.x -> milvux2.x : migrate_milvus2x_doc.
When migration finished, you can use visual tool Attu or use Milvus SDK verify your new collection data rows.
After the Milvus collection Data migration is completed, we can use SDK or Attu to create index and load collection for the next search operation.
- Support Redis to Milvus 2.x
- Support Mongodb to Milvus 2.x
- Support others datasource to Milvus 2.x
- Supports binary vectors
Contributions to milvus-migration are welcome from everyone. See Contributing for details of the contribution workflow.