[Bug]: [benchmark][standalone][LRU] search failed after reloading collection

[wangting0128]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: milvus-io-lru-dev-6e67e5a-20240411 
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task： lru-fouramf-txd8z

server:

NAME                                                              READY   STATUS                            RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lru-hanchun-etcd-0                                                1/1     Running                           0                 18h     10.104.31.125   4am-node34   <none>           <none>
lru-hanchun-milvus-standalone-5bdd8544f6-pkb9l                    1/1     Running                           1 (18h ago)       18h     10.104.31.124   4am-node34   <none>           <none>
lru-hanchun-minio-d6bbcf979-7kztl                                 1/1     Running                           0                 18h     10.104.31.122   4am-node34   <none>           <none>
lru-hanchun-pulsar-bookie-0                                       1/1     Running                           0                 18h     10.104.31.126   4am-node34   <none>           <none>
lru-hanchun-pulsar-bookie-1                                       1/1     Running                           0                 18h     10.104.32.246   4am-node39   <none>           <none>
lru-hanchun-pulsar-bookie-2                                       1/1     Running                           0                 3h31m   10.104.21.190   4am-node24   <none>           <none>
lru-hanchun-pulsar-bookie-init-29s4z                              0/1     Completed                         0                 18h     10.104.31.113   4am-node34   <none>           <none>
lru-hanchun-pulsar-broker-0                                       1/1     Running                           0                 18h     10.104.31.114   4am-node34   <none>           <none>
lru-hanchun-pulsar-proxy-0                                        1/1     Running                           0                 18h     10.104.9.4      4am-node14   <none>           <none>
lru-hanchun-pulsar-pulsar-init-bvtml                              0/1     Completed                         0                 18h     10.104.31.112   4am-node34   <none>           <none>
lru-hanchun-pulsar-recovery-0                                     1/1     Running                           0                 18h     10.104.31.111   4am-node34   <none>           <none>
lru-hanchun-pulsar-zookeeper-0                                    1/1     Running                           0                 18h     10.104.32.245   4am-node39   <none>           <none>
lru-hanchun-pulsar-zookeeper-1                                    1/1     Running                           0                 18h     10.104.31.128   4am-node34   <none>           <none>
lru-hanchun-pulsar-zookeeper-2                                    1/1     Running                           0                 3h47m   10.104.16.213   4am-node21   <none>           <none>

client pod name: lru-fouramf-txd8z-907689906
client log:

Expected Behavior

No response

Steps To Reproduce

1. create a collection, 3 fields: id(primaryKey), float_vector(768dim), int64_1
2. build HNSW index
3. insert 172000 data
4. flush collection
5. build index again with the same params
6. load collection
7. concurrent request: concurrent_num=1
   - load_search_release <- search raises error

Milvus Log

No response

Anything else?

client config:

{
     "dataset_params": {
          "dataset_name": "laion1b_nolang",
          "column_name": "float32_vector",
          "dim": 768,
          "dataset_size": "172000",
          "ni_per": 10000,
          "metric_type": "L2"
     },
     "collection_params": {
          "other_fields": [
               "int64_1"
          ],
          "shards_num": 1,
          "collection_name": "shard1_172000_512M"
     },
     "index_params": {
          "index_type": "HNSW",
          "index_param": {
               "M": 30,
               "efConstruction": 360
          }
     },
     "concurrent_tasks": [
          {
               "type": "load_search_release",
               "weight": 1,
               "params": {
                    "top_k": 1,
                    "nq": 10,
                    "search_param": {
                         "ef": 64
                    },
                    "timeout": null,
                    "random_data": true
               }
          }
     ],
     "concurrent_params": {
          "interval": 20,
          "during_time": "1h",
          "concurrent_number": 1
     }
}

test result:

[2024-04-11 13:00:17,736 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-04-11 13:00:17,737 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-04-11 13:00:17,737 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-11 13:00:17,737 -  INFO - fouram]: grpc     load_search_release                                                              709  708(99.86%) |   5071    3357    6700   5000 |    0.20        0.20 (stats.py:789)
[2024-04-11 13:00:17,737 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-11 13:00:17,737 -  INFO - fouram]:          Aggregated                                                                       709  708(99.86%) |   5071    3357    6700   5000 |    0.20        0.20 (stats.py:789)
[2024-04-11 13:00:17,737 -  INFO - fouram]:  (stats.py:790)
[2024-04-11 13:00:17,738 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': '',
            'deploy_mode': '',
            'config_name': '',
            'config': {},
            'host': 'lru-hanchun-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'column_name': 'float32_vector',
                                                    'dim': 768,
                                                    'dataset_name': 'laion1b_nolang',
                                                    'dataset_size': '172000',
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['int64_1'],
                                                       'shards_num': 1,
                                                       'collection_name': 'shard1_172000_512M'},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 30,
                                                                  'efConstruction': 360}},
                                 'concurrent_params': {'concurrent_number': 1,
                                                       'during_time': '1h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'load_search_release',
                                                       'weight': 1,
                                                       'params': {'top_k': 1,
                                                                  'nq': 10,
                                                                  'search_param': {'ef': 64},
                                                                  'timeout': None,
                                                                  'random_data': True}}]},
            'run_id': 2024041164091768,
            'datetime': '2024-04-11 11:53:29.096425',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 341.922},
                            'insert': {'total_time': 42.4198,
                                       'VPS': 4054.7103,
                                       'batch_time': 2.4658,
                                       'batch': 10000},
                            'flush': {'RT': 3.0169},
                            'load': {'RT': 3.5244},
                            'Locust': {'Aggregated': {'Requests': 709,
                                                      'Fails': 708,
                                                      'RPS': 0.2,
                                                      'fail_s': 1.0,
                                                      'RT_max': 6700.97,
                                                      'RT_avg': 5071.65,
                                                      'TP50': 5000.0,
                                                      'TP99': 6200.0},
                                       'load_search_release': {'Requests': 709,
                                                               'Fails': 708,
                                                               'RPS': 0.2,
                                                               'fail_s': 1.0,
                                                               'RT_max': 6700.97,
                                                               'RT_avg': 5071.65,
                                                               'TP50': 5000.0,
                                                               'TP99': 6200.0}}}}} 

[chyezh]

Old released segment is not cleared in lru cache.
So, after reloading, search by lru cache will access the released segment,
then segment not loaded is reported.

[chyezh]

@wangting0128 please verify with latest commit.

[wangting0128]

verification passed

argo task：lru-fouramf-wfklr
image：milvus-io-lru-dev-fbd02f7-20240418