milvus-2.4.5

v2.4.5

Release date: June 18, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.5	2.4.4	2.4.1	2.4.3

The release of Milvus 2.4.5 introduces several improvements and bug fixes to enhance performance, stability, and functionality. Milvus 2.4.5 simplifies sparse, float16, and bfloat16 vector search with auto-indexing, speeds up searches, deletions, and compactions with Bloom filter optimizations, and tackles data management through faster loading times and import L0 segment support. It also introduces the sparse HNSW index for efficient high-dimensional sparse data search, enhances the RESTful API with sparse float vector support, and fixes critical bugs for better stability.

Improvements

• Added RBAC support to describe/alter database API (#33804).
• Supported building the HNSW index for sparse vectors (#33653, #33662).
• Supported building the Disk index for binary vectors (#33575).
• Supported sparse vector type on RESTful v2 (#33555).
• Enabled flush rate limiter at the collection level (#33864).
• Executed Bloom filter application in parallel to speed up segment prediction (#33793).
• Used fastjson library for unmarshaling delete logs to speed up json.Unmarshal (#33802).
• Used BatchPkExist to reduce Bloom filter function call cost (#33752).
• Sped up the loading of small collections (#33746).
• Supported importing delete data to L0 segment (#33712).
• Skipped marking compaction tasks as timed out to avoid executing the same task repeatedly (#33833).
• Handled float16 and bfloat16 vectors similarly to BinaryVector in numpy bulk insert (#33788).
• Added the includeCurrentMsg flag for the seek method (#33743).
• Added mergeInterval, targetBufSize, and maxTolerantLag of msgdispatcher to configurations (#33680).
• Improved GetVectorByID for sparse vectors (#33652).
• Removed StringPrimarykey to reduce unnecessary copy and function call costs (#33649).
• Added autoindex mapping for binary/sparse data types (#33625).
• Optimized some caches to reduce memory usage (#33560).
• Abstracted execute interface for import/preimport tasks (#33607).
• Used map PK to timestamp in buffer insert to reduce Bloom filter causes (#33582).
• Avoided redundant meta operations during import (#33519).
• Improved logs by logging better disk quota info, adding UseDefaultConsistency flag, and removing unnecessary logs (#33597, #33644, #33670).

Bug fixes

• Fixed a bug that may cause Milvus to be unable to create AutoIndex on binary and sparse vectors (#33867).
• Fixed a bug that may cause indexnode to retry creating index on invalid index parameters of all vectors (#33878).
• Fixed a bug that may cause server crashes when loads and releases happen concurrently (#33699).
• Improved cache consistency for configuration values (#33797).
• Prevented possible data loss during deletion (#33821).
• Ensured the DroppedAt field (likely deletion timestamp) is set after dropping collections (#33767).
• Fixed an issue that might have caused Milvus to handle binary vector data sizes incorrectly (#33751).
• Masked Kafka credentials from being printed in logs (#33747).
• Prevented sensitive Kafka credentials from being logged in plain text (#33694).
• Ensured Milvus can correctly import data with multiple vector fields (#33724).
• Enhanced import reliability by checking if an import job exists before starting (#33673).
• Improved handling of the sparse HNSW index (internal functionality) (#33714).
• Cleaned vector memory to avoid memory leaks (#33708).
• Ensured smoother asynchronous warmup by fixing a state lock issue (#33687).
• Addressed a bug that might have caused missing results in query iterators (#33506).
• Fixed a bug that might cause import segment size to be uneven (#33634).
• Fixed incorrect data size handling for bf16, fp16, and binary vector types (#33488).
• Improved stability by addressing potential issues with the L0 compactor (#33564).
• Ensured dynamic configuration updates are reflected correctly in the cache (#33590).
• Improved the accuracy of the RootCoordQuotaStates metric (#33601).
• Ensured accurate reporting of the number of loaded entities in metrics (#33522).
• Provided more complete information in exception logs (#33396).
• Optimized query pipeline by removing unnecessary group checker (#33485).
• Used the local storage path for a more accurate disk capacity check on the index node (#33505).
• Corrected hasMoreResult to return false when hit number is larger than the limit (#33642).
• Delayed loading Bloom filters in delegator to prevent them from being loaded repeatedly when the worker has no more memory (#33650).

milvus-2.3.18

v2.3.18

Release date: June 18, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.18	2.3.7	2.3.6	2.3.6	2.3.5

Milvus v2.3.18 includes several bug fixes and improvements to enhance query result quality and maintainability. This release also backports improvements and fixes from the 2.4 branch, which enhance control over delete data amounts (according to rate limit configuration) and increase Bloom filter processing speed. As with previous releases, this update reinforces Milvus's commitment to functionality and robustness.

Improvements

• Added RESTful APIs to trigger component stop (#33798).
• Improved delete by expression to respect delete rate limit rules according to the matched data size instead of delete request size (#33794).
• Accelerated Bloom filter processing by submitting in batch and working in parallel (#33870).
• Enabled parallel processing for applydelete at the segment level (#33841).
• Printed UseDefaultConsistency parameter in read requests (#33645).

Bug fixes

• Fixed a bug where closure capture iteration variable caused delete data to fail to apply to the correct data (#33912).
• Fixed a bug where filtered results could be less than topk by upgrading Knowhere to 2.2.10 (#33930).
• Fixed a bug where insert rate was not limited when collection.insertrate.max.mb config was set to 0 (#33725).
• Fixed a bug where old collections created before v2.2.8 could reappear and become zombies after being dropped and restarted (#33695).
• Fixed a bug where upsert latency used the wrong unit, causing abnormal metrics (#33580).
• Fixed a bug where Milvus exception info could not be passed outside of segcore (#33395).

milvus-2.4.4

v2.4.4

Release date: May 31, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.4	2.4.3	2.4.1	2.4.2

Milvus v2.4.4 includes several critical bug fixes and improvements aimed at enhancing performance and stability. Notably, we've resolved a critical issue where bulk insert statistics logs were incorrectly garbage collected, potentially affecting data integrity. We strongly recommend all v2.4 users upgrade to this version to benefit from these fixes.

If you are using bulk insert, upgrade to v2.4.4 at the earliest opportunity for data integrity.

Critical bug fixes

• Filled stats log ID and validated its correctness (#33478)

Improvements

• Upgraded bitset for ARM SVE (#33440)
• Enabled Milvus compilation with GCC-13 (#33441)

Bug fixes

• Displayed empty collections when all privilege is granted (#33454)
• Ensured CMake downloads and installs for the current platform, not just x86_64 (#33439)

milvus-2.3.17

v2.3.17

Release date: May 31, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.17	2.3.7	2.3.6	2.3.6	2.3.5

Milvus version 2.3.17 primarily introduces several bug fixes to improve query accuracy, cluster reliability, and privilege maintenance. The most notable fix ensures that query iterators in Milvus return the correct number of results, rather than missing some data. This release reinforces Milvus's commitment to functionality and robustness.

Bug fixes

• Fixed issue where query iterators missed some results when upserts and deletes were performed very frequently (#33468)
• Fixed issue where the cluster could block during startup due to improper handling of standby status (#33371)
• Fixed issue where specifying the database name when granting privileges failed (#33292)

milvus-2.4.3

v2.4.3

Release date: May 29, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.3	2.4.3	2.4.1	2.4.2

Milvus version 2.4.3 introduced a host of features, improvements, and bug fixes to elevate performance and reliability. Notable enhancements included support for sparse float vector bulk insert and optimized bloom filter acceleration. Improvements covered various areas, from dynamic configuration updates to memory usage optimization. Bug fixes addressed critical issues like panic scenarios and ensured smoother system operations. This release underscored Milvus's ongoing commitment to enhancing functionality, optimizing performance, and delivering a robust user experience.

Features

• Supported sparse float vector bulk insert for binlog/json/parquet (#32649)

Improvements

• Implemented Datacoord/node watch channel based on RPC (#32036)
• Optimized bloom filter to accelerate delete filtering (#32642, #33329, #33284)
• Loaded raw data via mmap if scalar index did not have raw data (#33317)
• Synced milvus config to milvus.yaml (#33322, #32920, #32857, #32946)
• Updated knowhere version (#33310, #32931, #33043)
• Enabled dynamic updating of balancer policy in QueryCoord (#33272)
• Used a pre-built logger in the write buffer to minimize logger allocation (#33304)
• Improved parameter checking (#32777, #33271, #33218)
• Added a parameter to ignore incorrect message IDs in the checkpoint (#33249)
• Added config to control initialization failure handling for plugins (#32680)
• Added score compute consistency config for knowhere (#32997)
• Introduced a configuration option to control the initialization of public role permissions (#33174)
• Optimized memory usage when reading fields (#33196)
• Refined implementation of Channel Manager v2 (#33172, #33121, #33014)
• Added feature to track the size of data in memory for binlog (#33025)
• Added metrics for segment index files size (#32979, #33305)
• Replaced Delete with DeletePartialMatch to remove metrics (#33029)
• Got related data size according to segment type (#33017)
• Cleaned channel node info in meta store (#32988)
• Removed rootcoord from datanode broker (#32818)
• Enabled batch uploading (#32788)
• Changed default partition number to 16 when using partition key (#32950)
• Improved reduce performance on very large top-k queries (#32871)
• Utilized TestLocations ability to accelerate write & compaction (#32948)
• Optimized plan parser pool to avoid unnecessary recycling (#32869)
• Improved load speed (#32898)
• Used collection default consistency level for restv2 (#32956)
• Added cost response for the rest API (#32620)
• Enabled channel exclusive balance policy (#32911)
• Exposed describedatabase API in proxy (#32732)
• Utilized coll2replica mapping when getting RG by collection (#32892)
• Added more tracing for search & query (#32734)
• Supported dynamic config for opentelemetry trace (#32169)
• Avoided iteration over channel results when updating leaderview (#32887)
• Optimized vector offsets handling for parquet (#32822)
• Improved datacoord segment filtering with collection (#32831)
• Adjusted log level and frequency (#33042, #32838, #33337)
• Enabled stopping balance after balance had been suspended (#32812)
• Updated shard leader cache when leader location changed (#32470)
• Removed deprecated API and field (#32808, #32704)
• Added metautil.channel to convert string compare to int (#32749)
• Added type info for payload writer error message and log when querynode found new collection (#32522)
• Checked partition number when creating collection with partition key (#32670)
• Removed legacy l0 segment if watch failed (#32725)
• Improved printing type of request (#33319)
• Checked array field data was nil before getting the type (#33311)
• Returned error when startup Delete/AddNode node operation failed (#33258)
• Allowed datanode's server ID to be updated (#31597)
• Unified querynode metrics cleanup in collection release (#32805)
• Fixed scalar auto index config incorrect version (#32795)
• Refined index param check for create/alter index (#32712)
• Removed redundant replica recovery (#32985)
• Enabled channel meta table to write more than 200k segments (#33300)

Bug fixes

• Fixed panic when the database didn't exist in the rate limit interceptor (#33308)
• Fixed quotacenter metrics collection failure due to incorrect parameters (#33399)
• Fixed panic if processactivestandby returned an error (#33372)
• Fixed search result truncation in restful v2 when nq > 1 (#33363)
• Added database name field for role operations in restful v2 (#33291)
• Fixed global rate limit not working (#33336)
• Fixed panic caused by failure of building index (#33314)
• Added validation for sparse vector in segcore to ensure legality (#33312)
• Removed task from syncmgr after task completion (#33303)
• Fixed partition key filtering failure during data import (#33277)
• Fixed inability to generate traceID when using noop exporter (#33208)
• Improved query results retrieval (#33179)
• Marked channel checkpoint dropped to prevent checkpoint lag metrics leakage (#33201)
• Fixed query node getting stuck during stopping progress (#33154)
• Fixed missing segments in flush response (#33061)
• Made submit operation idempotent (#33053)
• Allocated new slice for each batch in streaming reader (#33360)
• Cleaned offline node from resource group after QueryCoord restart ([#33233](https://github.com...

v2.3.16

v2.3.16

Release date: May 24, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.16	2.3.7	2.3.6	2.3.6	2.3.5

Milvus v2.3.16 focuses on stability, performance, and security. It includes bug fixes for auto-balancing, checkpoint handling, memory leaks, and transaction limits. Additionally, authentication for the REST v2 API has been strengthened, and query result retrieval has been optimized. Overall, this update aims to provide a more reliable, efficient, and secure platform for your needs.

Improvements

• Enhanced authentication for the REST v2 API.(#33254)
• Improved checkpoint handling for message failures.(#33159)(#33128)
• Improved the triggering mechanism for segment sync tasks so that it will only be triggered once by flushts (#33034)
• Improved the query result retrieval mechanism to get enough query results (#33177)

Bug fixes

• Fixed a bug that when suspending auto balance querynode will unexpectedly exit stopping balance (#32941)
• Resolved an issue leading to endless binlog path downloads during compaction (#33246)
• Addressed memory leaks in rendezvousflushmanager (#33112)
• Prevented system panics related to nil array field data (#33119)
• Fixed "SegmentNotLoaded" errors triggered by concurrent segment tasks in search (#33087)
• Fixed a bug that Etcd txn exceeds limit due to too many fields (#33049)
• Fixed a bug related to the incorrect calculation of the milvus_rootcoord_indexed_entity_num metric (#33002)

milvus-2.3.15

v2.3.15

Release date: May 11, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.15	2.3.7	2.3.6	2.3.6	2.3.5

This release comes with several enhancements and bug fixes to improve security and stability. One of the improvements is the inclusion of the channel checkpoint in the flush response. Another enhancement is the ability to check whether autoID is enabled during data insertion, which optimizes the data migration using Milvus-CDC.

The bug fixes address issues such as empty segments in meta, and incorrect data types during data insertion according to the schema. A critical patch has also been implemented to solve the consistent failure and retries of the syncDistribution request to the delegator. We highly recommend updating to the latest version to benefit from these fixes.

Improvements

• Supported return channel checkpoint info in flush response (#32683)
• Add config to check id during inserting data when autoID enabled (#32840)

Critical Bug Fixes

• Fixing a bug where the syncDistribution request to the delegator consistently fails and retries persistently. (#32923)

Other Bug Fixes

• Fixed a bug where the system could generate empty segments. Added support for the compactor to clean up empty segments. (#32690)
• Fixed a bug that involved validating whether the input data types match the schema. (#32790, #32845)

milvus-2.4.1

v2.4.1

Release date: May 6, 2024

Milvus version	Python SDK version	Java SDK version	Node.js SDK version
2.4.1	2.4.1	2.4.0	2.4.2

Milvus version 2.4.1 brings numerous improvements and bug fixes that aim to enhance the software's performance, observability, and stability. These improvements include a declarative resource group API, enhanced bulk insert functionality that supports Float16/BFloat16 vector data types, a refined garbage collection (GC) mechanism that reduces list operations for object storage, and other changes related to performance optimizations. Additionally, bug fixes address issues such as compilation errors, failed fuzzy matches on newline characters, incorrect parameter datatypes for RESTful interfaces, and BulkInsert raising errors on numpy files when dynamic fields are enabled.

Breaking changes

• Discontinued support for delete with an empty filter expression. (#32472)

Features

• Added support for Float16/BFloat16 vector data types in bulk insert (#32157)
• Enhanced sparse float vector to support brute force iterator search and range search (#32635)

Improvements

• Added declarative resource group api (#31930 #32297 #32536 #32666)
• Rewrote the collection observer in QueryCoord to make it task-driven(#32441)
• Refactored the data structure used in the SyncManager of DataNode to reduce memory usage and prevent errors (#32673)
• Revised the implementation of garbage collection to minimize list operations associated with object storage(#31740)
• Reduced the cpu usage when collection number is high (#32245)
• Enhanced the management of milvus.yaml by automatically generating relevant configuration items in the milvus.yaml file through code(#31832 #32357)
• Enhanced the performance of the Query by retrieving the data after performing local reduction (#32346)
• Added WithBlock option for etcd client creation (#32641)
• Used client_request_id specified by the client as the TraceID if client provided(#32264)
• Added db label to the metrics for the delete and bulk insert operations(#32611)
• Added logic to skip the verification through configuration for AutoID and PartitionKey columns(#32592)
• Refined errors related to authentication (#32253)
• Refined error logs for AllocSegmentID in DataCoord(#32351 #32335)
• Removed duplicate metrics(#32380 #32308) and clean up unused metrics(#32404 #32515)
• Added configuration option to control whether to enforce the activation of the partitionKey feature(#32433)
• Added configuration option to control the maximum amount of data that can be inserted in a single request(#32433)
• Parallelize the applyDelete operation at the segment level to accelerate the processing of Delete messages by the Delegator(#32291)
• Used index (#32232 #32505 #32533 #32595) and add cache (#32580) to accelerate frequent filtering operations in QueryCoord.
• Rewrote the data structure(#32273) and refactor code(#32389 ) to accelerate common operations in DataCood.
• Removed openblas from conan(#32002)

Bug fixes

• Fixed build milvus in rockylinux8 (#32619)
• Fixed compilation errors for SVE on ARM (#32463 #32270)
• Fixed the crash issue on ARM-based GPU images(#31980).
• Fixed regex query can't handle text with newline (#32569)
• Fixed search get empty result caused by GetShardLeaders return empty node list (#32685)
• Fixed BulkInsert raised error when encountering dynamic fields in numpy files(#32596)
• Fixed bugs related to the RESTFulV2 interface, including an important fix that allows numeric parameters in requests to accept numerical input instead of string type(#32485 #32355)
• Fixed memory leak in proxy by remove watching config event in rate limiter(#32313)
• Fixed the issue where the rate limiter incorrectly reports that the partition cannot be found when partitionName is not specified(#32647)
• Added detection between the cases of Collection being in the recovery state and not being loaded in the error type.（#32447）
• Corrected the negative queryable num entities metric (#32361)

milvus-2.3.14

v2.3.14

Release date: Apr 29, 2024

Milvus version	Python SDK version	Java SDK version	Go SDK version	Node.js SDK version
2.3.14	2.3.7	2.3.6	2.3.6	2.3.5

This release focuses on improving system performance, reliability, and observability with key features. It supports RESTful APIs to execute rolling upgrades, implements task-driven collection observation, allows for configurable intervals for garbage collection (GC), and enhances the check logic.

Furthermore, it fixes bugs in RESTful v2, resolving issues such as parameter parsing errors and ineffective parameter handling. Critical bug fixes related to deadlocks and data accuracy have also been included. Additionally, performance improvements have been made when handling large numbers of segments.

Overall, these updates aim to provide a more secure, efficient, and reliable user experience.

Improvements

• Added configuration items to skip Auto ID and Partition Key checks to improve check speed.(#32671)
• Implemented the collection observer of QueryCoordV2 task-driven.(#32615)
• Supported the get sdk type by user agent in access log.(#32554)
• Supported the use of different intervals for GC scan.(#32551)
• Removed the support for the always-true expression in delete expr.(#32495)
• Allowed users to disable search optimization.(#32143)
• Added restful api for devops to execute rolling upgrade.(#31846)

Bug fixes

• Fixed the bug that the calculation precision may be different between growing index and sealed index.(#32584)
• Fixed the bug that grow task stuck when node become stopping.(#32556)
• Fixed the issue that leader view may not update when segment version did not match.(#32517)
• Fixed the bug that value in elementTypeParams not be integer in restful api.(#32362)
• Fixed dbName parameter of describeAlias request not take effect and fixed the issue which will report lack of permission when not pass the whole privileges to describe collection in restful api. (#32162)
• Fixed cp lag metrics leakage by marking channel checkpoint dropped.(#32454)
• Fixed the bug where not update segment's version in syncDistribution.(#32320)
• Fixed a bug that coordinator register blocked on ProcessActiveStandby.(#32133)
• Fixed a bug that the reduce didn't handle offset without limit and reduceStopForBest correctly.(#32087)
• Fixed a bug that not validata PlaceholderGroups before combine them.(#32045)
• Fixed a bug where acquire index meta's lock for each segment.(#31798)
• Fixed a bug that Rootcoord's stop may block in quota_center's stop.(#31824)

milvus-2.4.0

v2.4.0

Release date: April 17, 2024

Milvus version	Python SDK version
2.4.0	2.4.0

We are excited to announce the official launch of Milvus 2.4.0. Building upon the solid foundation of the 2.4.0-rc.1 release, we have focused on addressing critical bugs reported by our users, while preserving the existing functionality. In addition, Milvus 2.4.0 introduces a range of optimizations aimed at enhancing system performance, improving observability through the incorporation of various metrics, and streamlining the codebase for increased simplicity.

Improvements

• Support for MinIO TLS connections (#31396, #31618)
• AutoIndex support for scalar fields (#31593)
• Hybrid search refactoring for consistent execution paths with regular search (#31742, #32178)
• Accelerated filtering through bitset and bitset_view refactoring (#31592, #31754, #32139)
• Import tasks now support waiting for data index completion (#31733)
• Enhanced Import compatibility (#32121), task scheduling (#31475), and limits on imported file size and number (#31542).
• Code simplification efforts including interface standardization for type checking (#31945, #31857), removal of deprecated code and metrics (#32079, #32134, #31535, #32211, #31935), and normalization of constant names (#31515)
• New metrics for QueryCoord current target channel check point lag latency (#31420)
• New db label for common metrics(#32024)
• New metrics regarding the count of deleted, indexed, and loaded entities, with the inclusion of labels such as collectionName and dbName (#31861)
• Error handling improvements for mismatched vector types (#31766)
• Support for throwing errors instead of crashing when index cannot be built (#31845)
• Support for invalidating the database meta cache when dropping databases (#32092)
• Interface refactoring for channel distribution (#31814) and leader view management (#32127)
• Refactor channel dist manager interface(#31814) and Refactor leader view manager interface(#32127)
• Batch processing (#31632), adding mapping information (#32234, #32249), and avoiding usage of lock (#31787) to accelerate frequently invoked operations

Breaking Changes

• Discontinued grouping search on binary vectors (#31735)
• Discontinued grouping search with hybrid search(#31812)
• Discontinued HNSW index on binary vectors (#31883)

Bug Fixes

• Enhanced data type and value checks for queries and insertions to prevent crashes (#31478, #31653, #31698, #31842, #32042, #32251, #32204)
• RESTful API bug fixes (#32160)
• Improved prediction of inverted index resource usage (#31641)
• Resolution of connection issues with etcd when authorization is enabled (#31668)
• Security update for nats server (#32023)
• Stored inverted index files into a local storage path of QueryNode instead of /tmp (#32210)
• Addressed datacoord memory leaks for collectionInfo (#32243)
• Fixes for fp16/bf16 related bugs potentially causing system panic (#31677, #31841, #32196)
• Resolved issues with grouping search returning insufficient results (#32151)
• Adjustment of search with iterators to handle offsets in the Reduce step more effectively and ensure adequate results with "reduceStopForBest" enabled (#32088)