Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][cluster][LRU] search and query failed in dml & dql scene #32801

Closed
1 task done
wangting0128 opened this issue May 7, 2024 · 3 comments
Closed
1 task done

[Bug]: [benchmark][cluster][LRU] search and query failed in dml & dql scene #32801

wangting0128 opened this issue May 7, 2024 · 3 comments
Assignees
@jaime0815 @chyezh
Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.lru

Comments

@wangting0128
Copy link
Contributor

wangting0128 commented May 7, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: milvus-io-lru-dev-9234a94-20240506
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):  pulsar  
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: lru-fouramf-d5jpz

server:

NAME                                                              READY   STATUS        RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lru-verify-32135-cluster-etcd-0                                   1/1     Running       0               5m      10.104.19.173   4am-node28   <none>           <none>
lru-verify-32135-cluster-etcd-1                                   1/1     Running       0               5m      10.104.15.190   4am-node20   <none>           <none>
lru-verify-32135-cluster-etcd-2                                   1/1     Running       0               5m      10.104.18.232   4am-node25   <none>           <none>
lru-verify-32135-cluster-milvus-datacoord-5789ccf96f-4xk7b        1/1     Running       0               5m1s    10.104.24.132   4am-node29   <none>           <none>
lru-verify-32135-cluster-milvus-datanode-7447559b8b-2zgc9         1/1     Running       0               5m1s    10.104.24.135   4am-node29   <none>           <none>
lru-verify-32135-cluster-milvus-indexcoord-5d69fb9db8-2hgfl       1/1     Running       0               5m1s    10.104.24.136   4am-node29   <none>           <none>
lru-verify-32135-cluster-milvus-indexnode-6647c68789-5xb82        1/1     Running       0               5m1s    10.104.30.107   4am-node38   <none>           <none>
lru-verify-32135-cluster-milvus-indexnode-6647c68789-9tbh9        1/1     Running       0               5m1s    10.104.17.37    4am-node23   <none>           <none>
lru-verify-32135-cluster-milvus-proxy-767bf99dd-7flpv             1/1     Running       0               5m1s    10.104.24.131   4am-node29   <none>           <none>
lru-verify-32135-cluster-milvus-querycoord-8477cbc647-jvmrt       1/1     Running       0               5m1s    10.104.24.134   4am-node29   <none>           <none>
lru-verify-32135-cluster-milvus-querynode-856df586fb-tf77v        1/1     Running       0               5m1s    10.104.1.169    4am-node10   <none>           <none>
lru-verify-32135-cluster-milvus-rootcoord-7c78f55b49-f9qcv        1/1     Running       0               5m1s    10.104.24.130   4am-node29   <none>           <none>
lru-verify-32135-cluster-minio-0                                  1/1     Running       0               5m      10.104.19.172   4am-node28   <none>           <none>
lru-verify-32135-cluster-minio-1                                  1/1     Running       0               5m      10.104.15.191   4am-node20   <none>           <none>
lru-verify-32135-cluster-minio-2                                  1/1     Running       0               5m      10.104.24.142   4am-node29   <none>           <none>
lru-verify-32135-cluster-minio-3                                  1/1     Running       0               5m      10.104.18.233   4am-node25   <none>           <none>
lru-verify-32135-cluster-pulsar-bookie-0                          1/1     Running       0               5m      10.104.18.230   4am-node25   <none>           <none>
lru-verify-32135-cluster-pulsar-bookie-1                          1/1     Running       0               5m      10.104.19.174   4am-node28   <none>           <none>
lru-verify-32135-cluster-pulsar-bookie-2                          1/1     Running       0               5m      10.104.15.194   4am-node20   <none>           <none>
lru-verify-32135-cluster-pulsar-bookie-init-49mgf                 0/1     Completed     0               5m1s    10.104.24.133   4am-node29   <none>           <none>
lru-verify-32135-cluster-pulsar-broker-0                          1/1     Running       0               5m1s    10.104.6.6      4am-node13   <none>           <none>
lru-verify-32135-cluster-pulsar-proxy-0                           1/1     Running       0               5m1s    10.104.19.167   4am-node28   <none>           <none>
lru-verify-32135-cluster-pulsar-pulsar-init-njn79                 0/1     Completed     0               5m1s    10.104.24.140   4am-node29   <none>           <none>
lru-verify-32135-cluster-pulsar-recovery-0                        1/1     Running       0               5m1s    10.104.5.168    4am-node12   <none>           <none>
lru-verify-32135-cluster-pulsar-zookeeper-0                       1/1     Running       0               5m1s    10.104.18.229   4am-node25   <none>           <none>
lru-verify-32135-cluster-pulsar-zookeeper-1                       1/1     Running       0               4m22s   10.104.24.144   4am-node29   <none>           <none>
lru-verify-32135-cluster-pulsar-zookeeper-2                       1/1     Running       0               3m46s   10.104.15.196   4am-node20   <none>           <none> (base.py:257)
[2024-05-06 18:22:08,058 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'NAME|lru-verify-32135-cluster-milvus|lru-verify-32135-cluster-minio|lru-verify-32135-cluster-etcd|lru-verify-32135-cluster-pulsar|lru-verify-32135-cluster-zookeeper|lru-verify-32135-cluster-kafka|lru-verify-32135-cluster-log|lru-verify-32135-cluster-tikv'  (util_cmd.py:14)
[2024-05-06 18:22:19,029 -  INFO - fouram]: [CliClient] pod details of release(lru-verify-32135-cluster): 
 I0506 18:22:09.704805     517 request.go:665] Waited for 1.19818961s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/batch/v1beta1?timeout=32s
NAME                                                              READY   STATUS        RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lru-verify-32135-cluster-etcd-0                                   1/1     Running       0               12h     10.104.19.173   4am-node28   <none>           <none>
lru-verify-32135-cluster-etcd-1                                   1/1     Running       0               12h     10.104.15.190   4am-node20   <none>           <none>
lru-verify-32135-cluster-etcd-2                                   1/1     Running       0               12h     10.104.18.232   4am-node25   <none>           <none>
lru-verify-32135-cluster-milvus-datacoord-5789ccf96f-4xk7b        1/1     Running       0               12h     10.104.24.132   4am-node29   <none>           <none>
lru-verify-32135-cluster-milvus-datanode-7447559b8b-2zgc9         1/1     Running       0               12h     10.104.24.135   4am-node29   <none>           <none>
lru-verify-32135-cluster-milvus-indexcoord-5d69fb9db8-2hgfl       1/1     Running       0               12h     10.104.24.136   4am-node29   <none>           <none>
lru-verify-32135-cluster-milvus-indexnode-6647c68789-5xb82        1/1     Running       0               12h     10.104.30.107   4am-node38   <none>           <none>
lru-verify-32135-cluster-milvus-indexnode-6647c68789-9tbh9        1/1     Running       0               12h     10.104.17.37    4am-node23   <none>           <none>
lru-verify-32135-cluster-milvus-proxy-767bf99dd-7flpv             1/1     Running       0               12h     10.104.24.131   4am-node29   <none>           <none>
lru-verify-32135-cluster-milvus-querycoord-8477cbc647-jvmrt       1/1     Running       0               12h     10.104.24.134   4am-node29   <none>           <none>
lru-verify-32135-cluster-milvus-querynode-856df586fb-tf77v        1/1     Running       20 (25m ago)    12h     10.104.1.169    4am-node10   <none>           <none>
lru-verify-32135-cluster-milvus-rootcoord-7c78f55b49-f9qcv        1/1     Running       0               12h     10.104.24.130   4am-node29   <none>           <none>
lru-verify-32135-cluster-minio-0                                  1/1     Running       0               12h     10.104.19.172   4am-node28   <none>           <none>
lru-verify-32135-cluster-minio-1                                  1/1     Running       0               12h     10.104.15.191   4am-node20   <none>           <none>
lru-verify-32135-cluster-minio-2                                  1/1     Running       0               12h     10.104.24.142   4am-node29   <none>           <none>
lru-verify-32135-cluster-minio-3                                  1/1     Running       0               12h     10.104.18.233   4am-node25   <none>           <none>
lru-verify-32135-cluster-pulsar-bookie-0                          1/1     Running       0               12h     10.104.18.230   4am-node25   <none>           <none>
lru-verify-32135-cluster-pulsar-bookie-1                          1/1     Running       0               12h     10.104.19.174   4am-node28   <none>           <none>
lru-verify-32135-cluster-pulsar-bookie-2                          1/1     Running       0               12h     10.104.15.194   4am-node20   <none>           <none>
lru-verify-32135-cluster-pulsar-bookie-init-49mgf                 0/1     Completed     0               12h     10.104.24.133   4am-node29   <none>           <none>
lru-verify-32135-cluster-pulsar-broker-0                          1/1     Running       0               12h     10.104.6.6      4am-node13   <none>           <none>
lru-verify-32135-cluster-pulsar-proxy-0                           1/1     Running       0               12h     10.104.19.167   4am-node28   <none>           <none>
lru-verify-32135-cluster-pulsar-pulsar-init-njn79                 0/1     Completed     0               12h     10.104.24.140   4am-node29   <none>           <none>
lru-verify-32135-cluster-pulsar-recovery-0                        1/1     Running       0               12h     10.104.5.168    4am-node12   <none>           <none>
lru-verify-32135-cluster-pulsar-zookeeper-0                       1/1     Running       0               12h     10.104.18.229   4am-node25   <none>           <none>
lru-verify-32135-cluster-pulsar-zookeeper-1                       1/1     Running       0               12h     10.104.24.144   4am-node29   <none>           <none>
lru-verify-32135-cluster-pulsar-zookeeper-2                       1/1     Running       0               12h     10.104.15.196   4am-node20   <none>           <none>

client pod name: lru-fouramf-d5jpz-3796633683
client log:

[2024-05-06 13:05:44,194 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=65535, message=fail to Query on QueryNode 21: worker(21) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
[2024-05-06 13:06:02,560 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=65535, message=fail to search on QueryNode 21: worker(21) query failed:  => failed to load row ID or timestamp, potential missing bin logs or empty segments. Segment ID = 449570705034082594)>, <Time:{'RPC start': '2024-05-06 13:06:01.943796', 'RPC error': '2024-05-06 13:06:02.560093'}> (decorators.py:146)

test result:

[2024-05-06 18:20:48,147 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-05-06 18:20:48,147 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-05-06 18:20:48,147 -  INFO - fouram]: grpc     delete                                                                        562232   279(0.05%) |     40       1    5274     16 |   13.01        0.01 (stats.py:789)
[2024-05-06 18:20:48,147 -  INFO - fouram]: grpc     insert                                                                        561829   281(0.05%) |    219      23   12583    180 |   13.01        0.01 (stats.py:789)
[2024-05-06 18:20:48,147 -  INFO - fouram]: grpc     load                                                                          561432     0(0.00%) |     49       4    6101     25 |   13.00        0.00 (stats.py:789)
[2024-05-06 18:20:48,147 -  INFO - fouram]: grpc     query                                                                         560719 57974(10.34%) |    230       1  162124    110 |   12.98        1.34 (stats.py:789)
[2024-05-06 18:20:48,147 -  INFO - fouram]: grpc     search                                                                        561121 57551(10.26%) |    218       5  162326     91 |   12.99        1.33 (stats.py:789)
[2024-05-06 18:20:48,147 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-05-06 18:20:48,148 -  INFO - fouram]:          Aggregated                                                                   2807333 116085(4.14%) |    151       1  162326     82 |   64.98        2.69 (stats.py:789)
[2024-05-06 18:20:48,148 -  INFO - fouram]:  (stats.py:790)
[2024-05-06 18:20:48,150 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_8c16m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '2',
                                                              'memory': '8Gi',
                                                              'ephemeral-storage': '70Gi'},
                                                   'requests': {'cpu': '2',
                                                                'memory': '8Gi'}},
                                     'replicas': 1,
                                     'extraEnv': [{'name': 'LOCAL_STORAGE_SIZE',
                                                   'value': '70'}]},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '8Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '5Gi'}},
                                     'replicas': 2},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0',
                                                             'memory': '8Gi'},
                                                  'requests': {'cpu': '2.0',
                                                               'memory': '8Gi'}},
                                    'replicas': 1},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}},
                                 'persistence': {'size': '320Gi'}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'extraConfigFiles': {'user.yaml': 'queryNode:\n'
                                                         '  '
                                                         'diskCacheCapacityLimit: '
                                                         '51539607552\n'
                                                         '  mmap:\n'
                                                         '    mmapEnabled: '
                                                         'true\n'
                                                         '  lazyloadEnabled: '
                                                         'true\n'
                                                         '  '
                                                         'useStreamComputing: '
                                                         'true\n'
                                                         '  cache:\n'
                                                         '    warmup: sync\n'
                                                         '  '
                                                         'lazyloadWaitTimeout: '
                                                         '300000\n'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': 'milvus-io-lru-dev-9234a94-20240506'}}},
            'host': 'lru-verify-32135-cluster-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'column_name': 'float32_vector',
                                                    'dim': 768,
                                                    'scalars_params': {'int64_1': {'params': {'is_partition_key': True}}},
                                                    'dataset_name': 'laion1b_nolang',
                                                    'dataset_size': '10w',
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['int64_1'],
                                                       'num_partitions': 64},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 30,
                                                                  'efConstruction': 360}},
                                 'concurrent_params': {'concurrent_number': 10,
                                                       'during_time': '12h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 64,
                                                                  'timeout': 3000,
                                                                  'random_vector': True}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'delete_length': 64,
                                                                  'timeout': 3000}},
                                                      {'type': 'flush',
                                                       'weight': 0,
                                                       'params': {'timeout': 3000}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'timeout': 3000}},
                                                      {'type': 'search',
                                                       'weight': 1,
                                                       'params': {'top_k': 1,
                                                                  'nq': 10,
                                                                  'search_param': {'ef': 64},
                                                                  'expr': 'int64_1 '
                                                                          '>= '
                                                                          '0 '
                                                                          '&& '
                                                                          'int64_1 '
                                                                          '<= '
                                                                          '4',
                                                                  'timeout': 3000,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'expr': 'int64_1 '
                                                                          '!= '
                                                                          '2',
                                                                  'timeout': 3000,
                                                                  'offset': 0,
                                                                  'limit': 10}}]},
            'run_id': 2024050660597437,
            'datetime': '2024-05-06 06:14:19.839669',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 35.7288},
                            'insert': {'total_time': 25.8054,
                                       'VPS': 3875.1579,
                                       'batch_time': 2.5805,
                                       'batch': 10000},
                            'flush': {'RT': 5.1085},
                            'load': {'RT': 4.6199},
                            'Locust': {'Aggregated': {'Requests': 2807333,
                                                      'Fails': 116085,
                                                      'RPS': 64.98,
                                                      'fail_s': 0.04,
                                                      'RT_max': 162326.53,
                                                      'RT_avg': 151.49,
                                                      'TP50': 82,
                                                      'TP99': 800.0},
                                       'delete': {'Requests': 562232,
                                                  'Fails': 279,
                                                  'RPS': 13.01,
                                                  'fail_s': 0.0,
                                                  'RT_max': 5274.16,
                                                  'RT_avg': 40.11,
                                                  'TP50': 16,
                                                  'TP99': 370.0},
                                       'insert': {'Requests': 561829,
                                                  'Fails': 281,
                                                  'RPS': 13.01,
                                                  'fail_s': 0.0,
                                                  'RT_max': 12583.25,
                                                  'RT_avg': 219.05,
                                                  'TP50': 180.0,
                                                  'TP99': 1100.0},
                                       'load': {'Requests': 561432,
                                                'Fails': 0,
                                                'RPS': 13.0,
                                                'fail_s': 0.0,
                                                'RT_max': 6101.14,
                                                'RT_avg': 49.95,
                                                'TP50': 25,
                                                'TP99': 220.0},
                                       'query': {'Requests': 560719,
                                                 'Fails': 57974,
                                                 'RPS': 12.98,
                                                 'fail_s': 0.1,
                                                 'RT_max': 162124.56,
                                                 'RT_avg': 230.17,
                                                 'TP50': 110.0,
                                                 'TP99': 1100.0},
                                       'search': {'Requests': 561121,
                                                  'Fails': 57551,
                                                  'RPS': 12.99,
                                                  'fail_s': 0.1,
                                                  'RT_max': 162326.53,
                                                  'RT_avg': 218.43,
                                                  'TP50': 91,
                                                  'TP99': 1100.0}}}}}

Expected Behavior

No response

Steps To Reproduce

1. create a collection with 3 fields: id(primaryKey, autoID), float_vector(768dim), int64_1(partitionKey=64)
2. build HNSW index
3. prepare 10w data
4. flush collection
5. build index again with the same params
6. load collection
7. concurrent requests:
   - insert
   - delete
   - load
   - search
   - query

Milvus Log

No response

Anything else?

No response
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels May 7, 2024
@yanliang567 yanliang567 added this to the 2.4.lru milestone May 7, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 7, 2024
@yanliang567 yanliang567 removed their assignment May 7, 2024
@chyezh chyezh mentioned this issue May 7, 2024
Closed
@chyezh
Copy link
Contributor

chyezh commented May 7, 2024 •
edited
Loading

LoadIndex failure with an unexpected error which report creating directory failure, cannot be fixed in this turn.

[2024/05/06 12:34:55.352 +00:00] [WARN] [segments/load_index_info.go:231] ["CStatus returns err"] [traceID=c1245c727fbb577430a785d2c2f4d933] [error="filesystem error: cannot create directories: Not a directory [/var/lib/milvus/data/mmap/449570705034082594/101]"] [extra="AppendIndex failed"]

And segment_loader's error handling is broken, then assertion failure is_system_field_ready reported.
I will fix it, and wait the first bug reproduced, please don't release the cluster in next test.

@chyezh chyezh mentioned this issue May 9, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue May 10, 2024
@chyezh
fix: lru related issue fixup patch (#32916)
1c84a1c 
issue: #32206, #32801

- search failure with some assertion, segment not loaded and resource
insufficient.

- segment leak when query segments

---------

Signed-off-by: chyezh <chyezh@outlook.com>
@chyezh
Copy link
Contributor

chyezh commented May 10, 2024

is_system_field_ready failure is fixed, please verify it.
@wangting0128

UnyieldingOrca pushed a commit to UnyieldingOrca/milvus that referenced this issue May 10, 2024
@chyezh
fix: lru related issue fixup patch (milvus-io#32916)
cd183b7 
issue: milvus-io#32206, milvus-io#32801

- search failure with some assertion, segment not loaded and resource
insufficient.

- segment leak when query segments

---------

Signed-off-by: chyezh <chyezh@outlook.com>
UnyieldingOrca pushed a commit to UnyieldingOrca/milvus that referenced this issue May 10, 2024
@chyezh
fix: lru related issue fixup patch (milvus-io#32916)
8cd3b20 
issue: milvus-io#32206, milvus-io#32801

- search failure with some assertion, segment not loaded and resource
insufficient.

- segment leak when query segments

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: David Pichler <david.pichler@grainger.com>
@wangting0128
Copy link
Contributor Author

verification passed

image: master-20240514-f48a7ff8
argo task: lru-fouramf-wx7vx

test result:

{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'standalone',
            'config_name': 'standalone_8c16m',
            'config': {'standalone': {'resources': {'limits': {'cpu': '2',
                                                               'memory': '8Gi',
                                                               'ephemeral-storage': '70Gi'},
                                                    'requests': {'cpu': '2',
                                                                 'memory': '8Gi'}},
                                      'messageQueue': 'pulsar',
                                      'extraEnv': [{'name': 'LOCAL_STORAGE_SIZE',
                                                    'value': '70'}],
                                      'disk': {'size': {'enabled': True}}},
                       'cluster': {'enabled': False},
                       'etcd': {'replicaCount': 1,
                                'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'minio': {'mode': 'standalone',
                                 'metrics': {'podMonitor': {'enabled': True}},
                                 'persistence': {'size': '320Gi'}},
                       'pulsar': {'enabled': True},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'extraConfigFiles': {'user.yaml': 'queryNode:\n'
                                                         '  '
                                                         'diskCacheCapacityLimit: '
                                                         '51539607552\n'
                                                         '  mmap:\n'
                                                         '    mmapEnabled: '
                                                         'true\n'
                                                         '  lazyload:\n'
                                                         '    enabled: true\n'
                                                         '    waitTimeout: '
                                                         '300000\n'
                                                         '  '
                                                         'useStreamComputing: '
                                                         'true\n'
                                                         '  cache:\n'
                                                         '    warmup: off\n'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': 'master-20240514-f48a7ff8'}}},
            'host': 'lru-32987-dml-dql-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'column_name': 'float32_vector',
                                                    'dim': 768,
                                                    'scalars_params': {'int64_1': {'params': {'is_partition_key': True}}},
                                                    'dataset_name': 'laion1b_nolang',
                                                    'dataset_size': '10w',
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['int64_1'],
                                                       'num_partitions': 64},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 30,
                                                                  'efConstruction': 360}},
                                 'concurrent_params': {'concurrent_number': 10,
                                                       'during_time': '12h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 64,
                                                                  'timeout': 3000,
                                                                  'random_vector': True}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'delete_length': 64,
                                                                  'timeout': 3000}},
                                                      {'type': 'flush',
                                                       'weight': 0,
                                                       'params': {'timeout': 3000}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'timeout': 3000}},
                                                      {'type': 'search',
                                                       'weight': 1,
                                                       'params': {'top_k': 1,
                                                                  'nq': 10,
                                                                  'search_param': {'ef': 64},
                                                                  'expr': 'int64_1 '
                                                                          '>= '
                                                                          '0 '
                                                                          '&& '
                                                                          'int64_1 '
                                                                          '<= '
                                                                          '4',
                                                                  'timeout': 3000,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'expr': 'int64_1 '
                                                                          '!= '
                                                                          '2',
                                                                  'timeout': 3000,
                                                                  'offset': 0,
                                                                  'limit': 10}}]},
            'run_id': 2024051412144173,
            'datetime': '2024-05-14 12:53:34.093352',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 63.8284},
                            'insert': {'total_time': 27.8717,
                                       'VPS': 3587.8687,
                                       'batch_time': 2.7872,
                                       'batch': 10000},
                            'flush': {'RT': 4.9282},
                            'load': {'RT': 3.5367},
                            'Locust': {'Aggregated': {'Requests': 624242,
                                                      'Fails': 0,
                                                      'RPS': 14.45,
                                                      'fail_s': 0.0,
                                                      'RT_max': 68522.33,
                                                      'RT_avg': 689.36,
                                                      'TP50': 98,
                                                      'TP99': 15000.0},
                                       'delete': {'Requests': 124461,
                                                  'Fails': 0,
                                                  'RPS': 2.88,
                                                  'fail_s': 0.0,
                                                  'RT_max': 20965.18,
                                                  'RT_avg': 48.13,
                                                  'TP50': 34,
                                                  'TP99': 280.0},
                                       'insert': {'Requests': 124810,
                                                  'Fails': 0,
                                                  'RPS': 2.89,
                                                  'fail_s': 0.0,
                                                  'RT_max': 20390.22,
                                                  'RT_avg': 348.05,
                                                  'TP50': 260.0,
                                                  'TP99': 1800.0},
                                       'load': {'Requests': 125020,
                                                'Fails': 0,
                                                'RPS': 2.89,
                                                'fail_s': 0.0,
                                                'RT_max': 20068.64,
                                                'RT_avg': 75.72,
                                                'TP50': 53,
                                                'TP99': 430.0},
                                       'query': {'Requests': 125121,
                                                 'Fails': 0,
                                                 'RPS': 2.9,
                                                 'fail_s': 0.0,
                                                 'RT_max': 68510.11,
                                                 'RT_avg': 1502.58,
                                                 'TP50': 130.0,
                                                 'TP99': 24000.0},
                                       'search': {'Requests': 124830,
                                                  'Fails': 0,
                                                  'RPS': 2.89,
                                                  'fail_s': 0.0,
                                                  'RT_max': 68522.33,
                                                  'RT_avg': 1469.38,
                                                  'TP50': 110.0,
                                                  'TP99': 23000.0}}}}}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jaime0815 jaime0815

@chyezh chyezh

Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.lru
Development

No branches or pull requests
4 participants
@jaime0815 @chyezh @wangting0128 @yanliang567