Milvus response time for vector search fluctuatesrapidly

[Pavan29082003]

I am experiencing significant fluctuations in query times while running Milvus on an EC2 instance using Docker. Initially, query times are fast (around 1000ms), but they occasionally spike to over 9s, and this happens inconsistently during normal operations. The issue becomes more pronounced as the data volume increases.

[Pavan29082003]

I am hosting milvus on ec2 using this docker file

ersion: '3.5'

services:
etcd:
container_name: milvus-etcdd
image: quay.io/coreos/etcd:v3.5.5
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd

minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2023-03-20T20-16-18Z
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.4.0
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- "etcd"
- "minio"
attu:
container_name: attu
image: zilliz/attu:v2.2.6
environment:
MILVUS_URL: milvus-standalone:19530
ports:
- "8000:3000"
depends_on:
- "standalone"

networks:
default:
name: milvus

[Pavan29082003]

This the memory usage of all containers that are in my ec2.There is no fluctuation in this data while I am querying from milvus

[yhmo]

It sounds like your use case is to continually insert data and query.
Some questions:

1. Which version of the milvus?
2. How many entities are in the collection? Could you show me the collection's schema?
3. Averagely how many entities are inserted into the collection per second?
4. Which interface fluctuates? collection.search() or collection.query()? Did you set "consistency_level"? Did you set "output_fields"?

[yhmo]

The time cost mainly depends on the length of the JSON content. Maybe the "body_content" of some rows is simple then the query is faster.

[Pavan29082003]

I have been using same input query vector throughout the day.So the results that are being returned are all same each and every time

[yhmo]

A workaround you can try:

1. create another three collections, the same schema, except the "body_content" is a Varchar field
2. in client side, use json.dumps() to convert the body_content to a str, and insert into the new collections.
3. query in the same way, output_fields="*"
4. in client side, use json.loads() to convert the body_content string to a json object.

I am not sure how much faster is the workaround, I need to do some local tests to verify this performance issue.

[Pavan29082003]

Removing body_content from output fields made a lot of differnece.Now I am able to query in just 500 ms.Thank you so much for identifyng the problem

[yhmo]

@Pavan29082003
I just used your script to test in my local.
100000 entities for each collection with the "body_content" you provided.
The average time cost is no more than 100ms per search. I guess the bottleneck is in your network bandwidth.