Milvus Roadmap 2025

[yiwen92]

Milvus 2025 Roadmap Discussion

Objective: Facilitate community collaboration to prioritize features and collect scenario-specific feedback.
How to Participate: Comment on specific features with your use cases or implementation suggestions, please feel free to use reactions (👍 for high priority) to indicate the importance of each feature.

---

Category 1: AI-Driven Unstructured Data Processing

• Data-In/Out: Native integration with major model services for raw text ingestion
• Original Data Handling: Text/URL reference support for raw data processing
• Tensor Support: Vector list implementation (for ColBERT/CoPali/Video scenarios)
• Extended Data Types: DateTime, Map, GIS support based on requirements
• Iterative Search: Query vector refinement through user‘s feedback

---

Category 2: Search Quality & Performance

• Advanced Matching: phrase_match & multi_match capabilities
• Analyzer Upgrade: Enhance Analyzer with expanded tokenizer support and improved observability
• JSON Optimization: Faster filtering through improved indexing
• Execution Sorting: Scalar field-based result ordering
• Advanced Reranker: Model-based reranking & custom scoring functions
• Iterative Search: Query vector refinement through user‘s feedback

---

Category 3: Rich Functionality & Management

• Schema Change: Add/delete field, modify varchar length
• Scalar Aggregations: count/distinct/min/max operations
• Support UDF: Support user defined function
• Data Versioning: Snapshot-based rollback system
• Data Clustering: Co-location through configuration
• Data Sampling: Fast get results based on sampling data

---

Category 4: Architecture Upgrade & Easy Deployment

• Stream Node: Simplified incremental data ingestion
• MixCoord: Unified coordinator architecture
• Logstore Independence: Reduced external dependencies like pulsar
• PK Deduplication: Global primary key deduplication

---

Category 5: Vector Lake Solution & Cost Efficiency

• Tiered Storage: Hot/cold data separation for lower storage cost
• Data Evict Policy: Users can define their own data evict policy
• Bulk Updates: Support field-specific value modifications, ETL etc
• Large TopK: Return massive datasets
• VTS GA: Connect to different sources of data
• Advanced Quantization: Optimize memory consumption and performance based on quantization techniques
• Resource Elasticity: Dynamically scale resources to accommodate varying write loads, read loads, and background task loads

---

Note: We will carefully listen to your feedback and appropriately adjust and iterate our roadmap, especially the features planned for Milvus 3.0.

[Niceies]

User experience

1. There are many basic parameters for deployment. It took me an afternoon to deploy the milvus cluster for the first time (with basic helm and basic deployment of other components). I only configured basic parameters (including milvus and other dependent components): cpu, memory, disk, auth, addressport, etc., up to 120. The more parameters, the easier it is to make mistakes. Of course, you can also use a layer of deployment plug-in to maintain the deployment scenario and fine-tune, but if the deployment of milvus itself is relatively simple, you don't need this auxiliary tool.
2. The compilation environment is quite strict. I tried to install the build guide to build the deployment environment, but various errors (related to version dependency) kept appearing during the compilation process. It should be a problem with the dependency on the C library. In the end, I could only compile successfully through the compilation image provided by the official. In addition, the dependent library has high requirements for external network connectivity, which should be caused by the reference of more C libraries. When I compiled on the intranet (limited by the office environment), I needed to open proxy sources such as go and github, and modify the download address in the code. Some C addresses were hard-coded and could not be modified. I could only replace the proxy source by modifying the python download code. I built the rust library on the intranet to succeed.
3. The community releases versions frequently and is in a stage of rapid iteration. From the announcements, many of them are about improving stability and reliability, and need to be upgraded as soon as possible. However, for some users, frequent upgrades are not possible. We hope to ensure stability and reliability as much as possible when developing new functions and iterations.
4. Are the related versions of milvus-backup, birdwatch, milvus-helm, and milvus-operator related? Sometimes, when using the backup function, deployment, or debugging with the wrong version, the function will fail and you need to try to adjust it back and forth.

Subsequent use plan

1. Continuously update the community version
2. Test the use of mmap to reduce performance while meeting performance requirements
3. Compare es's full-text search capabilities and scenarios to decide whether to replace es

Summary
I hope to use a database that is simpler, more user-friendly, more reliable, and better than similar databases in terms of core capabilities such as vector search and full-text search.

[yiwen92]

Thank you for your feedback.

1. First, we recommend using [this sizing tool]to estimate capacity and obtain a default configuration file, so you don’t need to configure everything manually. Additionally, we do have plans to simplify Milvus deployment management and dependencies, which is also reflected in our roadmap. Lastly, we plan to release best practices and more granular guidance on deployment and parameter configuration in the near future, either as a blog post or documentation.

2. We will work on optimizing it.

3. Before each release, we conduct thorough testing to ensure overall stability and compatibility. However, unforeseen bugs may still arise, which we will address through future releases. We recommend that community users follow our latest versions at an appropriate frequency to benefit from new features and optimizations.

4. Please refer to our release notes, where we provide details on compatibility.

[xiaofan-luan]

User experience

1. There are many basic parameters for deployment. It took me an afternoon to deploy the milvus cluster for the first time (with basic helm and basic deployment of other components). I only configured basic parameters (including milvus and other dependent components): cpu, memory, disk, auth, addressport, etc., up to 120. The more parameters, the easier it is to make mistakes. Of course, you can also use a layer of deployment plug-in to maintain the deployment scenario and fine-tune, but if the deployment of milvus itself is relatively simple, you don't need this auxiliary tool.
2. The compilation environment is quite strict. I tried to install the build guide to build the deployment environment, but various errors (related to version dependency) kept appearing during the compilation process. It should be a problem with the dependency on the C library. In the end, I could only compile successfully through the compilation image provided by the official. In addition, the dependent library has high requirements for external network connectivity, which should be caused by the reference of more C libraries. When I compiled on the intranet (limited by the office environment), I needed to open proxy sources such as go and github, and modify the download address in the code. Some C addresses were hard-coded and could not be modified. I could only replace the proxy source by modifying the python download code. I built the rust library on the intranet to succeed.
3. The community releases versions frequently and is in a stage of rapid iteration. From the announcements, many of them are about improving stability and reliability, and need to be upgraded as soon as possible. However, for some users, frequent upgrades are not possible. We hope to ensure stability and reliability as much as possible when developing new functions and iterations.
4. Are the related versions of milvus-backup, birdwatch, milvus-helm, and milvus-operator related? Sometimes, when using the backup function, deployment, or debugging with the wrong version, the function will fail and you need to try to adjust it back and forth.

Subsequent use plan

1. Continuously update the community version
2. Test the use of mmap to reduce performance while meeting performance requirements
3. Compare es's full-text search capabilities and scenarios to decide whether to replace es

Summary I hope to use a database that is simpler, more user-friendly, more reliable, and better than similar databases in terms of core capabilities such as vector search and full-text search.

Hi Niceies,

1. Milvus is a cloud native and cloud only system. The recommended way to run it is always using Zilliz cloud or host it in K8s with our image. If you can use K8s Operator or Helm your life will be much easier.

2. We'd like to take issues and contributions(Better) if possible. We have some experts in the community and they'd like to improve.

3. 2.4 branch is pretty stable and should be able to serve for quite well. 2.5/2.6 branch is actively under heavy development. If you are not a full text search, right not 2.4.23 works perfectly.

4. Good suggetion. I will definitely take this one.

If you have time I'd definitely like to meet, offer some help and receive more feedback to help you to achieve your target.

Feel free to reach me at james.luan@zilliz.com. I'd like to take an appointment from anyone else who has feedback for the community

[HuaJieHappy]

Schema Change: Add/delete field, modify varchar length
Bulk Updates: Support field-specific value modifications, ETL etc
在B端场景的RAG中，会遇到需要迁移数据/批量修改某个字段的值/新增、修改、删除某个字端，现阶段只能通过脚本把数据导出到新collection中，很需要这些功能的支持

[xiaofan-luan]

Schema Change: Add/delete field, modify varchar length Bulk Updates: Support field-specific value modifications, ETL etc 在B端场景的RAG中，会遇到需要迁移数据/批量修改某个字段的值/新增、修改、删除某个字端，现阶段只能通过脚本把数据导出到新collection中，很需要这些功能的支持

Add column/Delete column is already on the way.

We may opensource our special source modify schema using spark connector soon