High Memory consumption on query node

[Satyadev592]

Milvus Version: 2.2.9
Number of entries: 115 Million
Dimesnion: 256
Index: HNSW
Query Nodes: 19 nodes with 64GB RAM and 8 cores CPU.

We have been using milvus for over a year in production with minimal issues. We have always seen query node memory increasing over a period of few weeks. We restarted the query nodes to resolve this and were doing this periodically. Recently we added a new collection to the same cluster (although a much smaller collection) ~ 4-5 Million entries (~4% size of the other collection). We are noticing frequent memory spikes as a result of this.

All graphs are for last 1 day:

Query Node Memory Utilisation:

Segment Number by Segment Stage Graph:

Kindly help us debug and resolve this.

[xiaofan-luan]

is there any insertions or deletion on that new collection?

[yhmo]

Seems the "milvus_querynode_segment_num(*) by {segment_state}" counts all segments including dropped/growing/sealed. You can specify the segment_state respectively to observe the number of different states of segments.

[Satyadev592]

The above graph contains two lines - one for sealed and the other for growing. Uploading image of the same graph with both lines clearly highlighted. I don't however see any info on deleted.

[yhmo]

How many partitions in the collection? 400?

[Satyadev592]

Also, do note that the number of segments decreasing from 1200 to 700 is because we issued a restart of all query nodes. I am not sure if your analysis that it's because of compaction holds true there. The number of partitions is 504.

[yhmo]

Could you show me the "flushed" and "dropped" segments curve?

[yhmo]

115M 256dim, the original data size is 115GB.
HNSW index size is a bit larger than the original data size.
Ideally, the memory consumption of each query node should be 115/19 = 6GB.
From the screenshot, seems each query node's memory usage is more than 30GB.

Did you set replica_num when you call the load() interface?
Are there any scalar fields with long-length varchar in the collection?

[Satyadev592]

There are no scalar fields, just the id and embedding. Yes, replica number is set to 2.