Performance: Slowdown When Increasing Search Limit

[dmitriivahrushev]

Hello everyone!

I need some help with a performance issue I'm encountering, please.

I have a collection with the following structure:

{
    "code": 0,
    "data": {
        "aliases": [],
        "autoId": false,
        "collectionID": 463164325200582997,
        "collectionName": "m16_ef1024_2025_12_25_55M",
        "consistencyLevel": "Bounded",
        "description": "",
        "enableDynamicField": false,
        "fields": [
            {
                "autoId": false,
                "clusteringKey": false,
                "description": "",
                "id": 100,
                "name": "detect_id",
                "nullable": false,
                "partitionKey": false,
                "primaryKey": true,
                "type": "Int64"
            },
            {
                "autoId": false,
                "clusteringKey": false,
                "description": "",
                "id": 101,
                "name": "vector",
                "nullable": false,
                "params": [
                    {
                        "key": "dim",
                        "value": "512"
                    }
                ],
                "partitionKey": false,
                "primaryKey": false,
                "type": "FloatVector"
            }
        ],
        "functions": [],
        "indexes": [
            {
                "fieldName": "vector",
                "indexName": "vector",
                "metricType": "COSINE"
            }
        ],
        "load": "LoadStateLoaded",
        "partitionsNum": 1,
        "properties": [
            {
                "key": "timezone",
                "value": "UTC"
            }
        ],
        "shardsNum": 1
    },
    "message": ""
}

When performing vector searches using pymilvus's search() method, I'm observing significant performance degradation as I increase the limit parameter:

Performance measurements:
limit=100 → 580 vectors per second
limit=500 → 141 vectors per second (4x slowdown)
limit=5000 → 20 vectors per second (29x slowdown)

My question:
Are such dramatic performance drops normal/expected behavior when increasing the result limit?

I would greatly appreciate any insights, explanations, or optimization suggestions you might have!
Thank you in advance for your help!

[yhmo]

Possible reason: the search() request returns vector field in the "output_fields"

results = client.search(collection_name="XXX",
                             data=[vector_to_search],
                             limit=10,
                             output_fields=["vector"]
                             )

By default, vector field is not loaded into memory(only the index is loaded into querynode's memory). If a search request retrieves a vector field by "output_fields", it lazily reads the vectors from disk cache and search latency increases.
This behavior is configured in the milvus.yaml.
In v2.6.x, the queryNode.tieredStorage.warmup.vectorField is "disable":

queryNode:
    tieredStorage:
      warmup:
        # options: sync, disable.
        # Specifies the timing for warming up the Tiered Storage cache.
        # - "sync": data will be loaded into the cache before a segment is considered loaded.
        # - "disable": data will not be proactively loaded into the cache, and loaded only if needed by search/query tasks.
        # Defaults to "sync", except for vector field which defaults to "disable".
        scalarField: sync
        scalarIndex: sync
        vectorField: disable # cache warmup for vector field raw data is by default disabled.
        vectorIndex: sync

[dmitriivahrushev]

Thank you very much!!!