Best Practices for Milvus Schema Migrations Across Environments

[Kevinpaulson2002]

Hi everyone,

I'm a beginner with Milvus and would like to understand the recommended best practices for managing collection schemas in a production application.

Suppose Tool Mapper Service v1 in QA uses a collection with schema v1. When we deploy Tool Mapper Service v1 to UAT and production, the collection schema should remain consistent. Later, in QA v2, we introduce a schema change, such as adding a field or changing the vector configuration. How should we safely propagate that change to UAT and production?

In relational databases, we use tools like Flyway to manage versioned schema migrations (after altering or dropping the columsn or any table schema changes). Is there a similar recommended tool or standard practice for Milvus?

Also, can collection creation and schema modifications be managed through an SDK or CLI, or are we expected to use REST APIs/cURL?

I'd appreciate any guidance on the recommended workflow for managing schema versions and migrations across QA, UAT, and production.

Thank you!

[yhmo]

Hi @Kevinpaulson2002,
Great questions. Here is an answer based on Milvus 3.0 (the latest major version)

1. Online schema changes

Milvus 3.0 introduced online schema evolution: you can add, backfill, and drop columns while the collection keeps serving traffic — no offline rebuild of the collection is required.

Change	API	Notes
Add a scalar field	add_collection_field() / REST POST /v2/vectordb/collections/fields/add	New field must be nullable=True; existing entities get NULL (or a default_value you provide). Takes effect almost immediately.
Add a vector field	add_collection_field()	Also supported in Milvus 3.0 thanks to nullable vector fields: the added vector field must be nullable=True. Existing entities have NULL for it until you write vector values (via upsert or an external backfill path); then create an index on the new field for ANN search. For function-generated fields (BM25 sparse vectors, MinHash), use add_function_field(), which atomically commits field + function + index params.
Drop a field	drop_collection_field()	Works for user-defined scalar/vector/StructArray fields that are no longer needed; it updates collection metadata at runtime.
Alter field properties	alter_collection_field()	e.g. VARCHAR max_length, ARRAY max_capacity, mmap.enabled.
Backfill new values	inner backfill / external backfill	Inner backfill (GA in 3.0) computes values from data already in the collection (e.g. attach a BM25/MinHash function to generate sparse vectors for existing text). External backfill handles values computed outside Milvus (e.g. via snapshots + Spark).

For example:

from pymilvus import MilvusClient, DataType

client = MilvusClient(uri="http://localhost:19530")

# add a nullable field with a default for existing rows
client.add_collection_field(
    collection_name="tool_mapper",
    field_name="status",
    data_type=DataType.VARCHAR,
    max_length=32,
    nullable=True,
    default_value="active",
)

# drop a deprecated field
client.drop_collection_field(
    collection_name="tool_mapper",
    field_name="legacy_tag",
)

# add a new (nullable) embedding column alongside the old one
client.add_collection_field(
    collection_name="tool_mapper",
    field_name="embedding_v2",
    data_type=DataType.FLOAT_VECTOR,
    dim=768,
    nullable=True,  # required: existing rows are NULL until backfilled
)

What still cannot be changed in place:

• You cannot change an existing vector field's type or dimension in place. To migrate embeddings, add a new nullable vector field (as above), backfill it via upsert, build its index, switch searches to it, and drop the old field — no collection rebuild needed.
• The primary key and partition key are fixed once the collection is created.

So for your example: both "adding a field" and even "adding a new vector column" are online operations in 3.0. A full collection rebuild is only required for changes that cannot be expressed as a new field (e.g., changing the primary key or partition key).

2. Is there a Flyway equivalent for Milvus?

There is no official Flyway/Liquibase-style migration tool for Milvus today. The milvus-migration tool is for migrating data from Milvus 1.x to 2.x, not for schema version management.

The common practice is to build the same workflow yourself, which is straightforward since all schema operations are SDK/REST calls:

1. Schema as code: keep numbered migration scripts in your service repo, e.g.

   migrations/
   ├── V1__create_tool_mapper.py     # create collection (schema v1)
   ├── V2__add_status_field.py       # add nullable status field
   └── V3__drop_legacy_tag.py        # drop deprecated field
   

2. Record applied versions: keep a small history table (in MySQL/Postgres) or store the current schema version as a collection property via alter_collection() (e.g., schema.version = 3) so each environment knows which migrations have been applied.
3. Apply on deploy: during deployment, compare the environment's applied version against the repo and run the pending scripts in order. The same scripts promote QA → UAT → production through your CI/CD pipeline.
4. Never edit an applied migration; always add a new numbered one (exactly like Flyway).

For truly breaking changes that cannot be expressed as a new nullable field — e.g., changing the primary key or partition key — use a blue/green pattern since these can't be altered in place:

1. Create tool_mapper_v2 with the new schema.
2. Backfill it from the old collection (Milvus backup/restore, snapshots + Spark DataSource V2, or bulk export/import).
3. Validate results in QA, then dual-write or do a final delta sync in higher environments.
4. Switch readers via a collection alias (tool_mapper → tool_mapper_v2), then retire the old collection after a safety window.

3. SDK, CLI, or REST?

All of them are available — REST/cURL is not required:

• SDKs (recommended for migration scripts): Python (MilvusClient), Java (MilvusClientV2), Node.js, Go, and Rust all support create collection, add/drop field, and alter field operations in 3.0.
• RESTful v2 API: the same operations are exposed as HTTP endpoints, e.g. POST /v2/vectordb/collections/fields/add.
• Milvus_CLI: supports connecting, creating/dropping collections, indexes, partitions, and import/export. The schema-evolution operations themselves are usually driven through an SDK or REST inside your migration runner.
• Attu (GUI) is handy for visually inspecting collections across environments.

Suggested workflow summary

1. Define the schema and all migrations in code, next to the Tool Mapper Service.
2. On deploy, the pipeline applies only the pending numbered migrations per environment, and records the new schema version.
3. Use online add/drop/alter for additive schema changes (3.0 supports them while serving).
4. Embedding-model changes usually fit the online path: add a new nullable vector field, backfill, index, and switch searches to it. Keep versioned blue/green collections + alias switch for changes to the primary key, partition key, or storage-level rebuilds.
5. Avoid manual console changes in UAT/production — everything should be reproducible from the migration scripts.

Relevant docs:

• Alter Collection Schema: https://milvus.io/docs/add-fields-to-an-existing-collection.md
• Alter Collection Field: https://milvus.io/docs/alter-collection-field.md
• Milvus 3.0 announcement (online schema evolution & backfill): https://milvus.io/blog/announcing-milvus-3-lake-native-vector-search-and-a-more-powerful-retrieval-engine.md
• Milvus_CLI command reference: https://milvus.io/docs/cli_commands.md

Hope this helps