Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][cluster] Load collection raises error show collection failed: Assert "insert_record_.empty_pks() in load_release and dml scenario #30350

Closed
1 task done
wangting0128 opened this issue Jan 30, 2024 · 7 comments
Closed
1 task done

[Bug]: [benchmark][cluster] Load collection raises error show collection failed: Assert "insert_record_.empty_pks() in load_release and dml scenario #30350

wangting0128 opened this issue Jan 30, 2024 · 7 comments
Assignees
@zhagnlu
Labels
2.4-features kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.0

Comments

@wangting0128
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master-20240129-927d3109-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.0rc19
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: multi-vector-scene-mix-cgbbg
test case name: test_hybrid_search_locust_dml_load_release_cluster

server:

[2024-01-29 06:57:25,337 -  INFO - fouram]: [Base] Deploy initial state: 
I0129 03:55:31.833374     448 request.go:665] Waited for 1.175263815s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/autoscaling/v2?timeout=32s
NAME                                                              READY   STATUS        RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-scene-mix-cgbbg-3-etcd-0                             1/1     Running       0               16m     10.104.33.60    4am-node36   <none>           <none>
multi-vector-scene-mix-cgbbg-3-etcd-1                             1/1     Running       2 (9m43s ago)   16m     10.104.15.154   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-etcd-2                             1/1     Running       0               16m     10.104.26.86    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-datacoord-746f8f464dc6r65   1/1     Running       2 (7m37s ago)   16m     10.104.26.24    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-datanode-c9fd959f9-4p4gc    1/1     Running       3 (3m37s ago)   16m     10.104.29.159   4am-node35   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-indexcoord-f56df7db7mj777   1/1     Running       0               16m     10.104.15.114   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-indexnode-5898675f6-bmpql   1/1     Running       1 (11m ago)     16m     10.104.26.25    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-proxy-dd77f56f9-7m68p       1/1     Running       3 (2m37s ago)   16m     10.104.26.27    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-querycoord-564478f47flq4c   1/1     Running       3 (3m36s ago)   16m     10.104.26.28    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-querynode-6bb7ff6c56bp2x5   1/1     Running       2 (8m4s ago)    16m     10.104.33.250   4am-node36   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-rootcoord-6f6d9f55fbxpvkt   1/1     Running       3 (5m37s ago)   16m     10.104.26.26    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-minio-0                            1/1     Running       0               16m     10.104.15.158   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-minio-1                            1/1     Running       0               16m     10.104.33.59    4am-node36   <none>           <none>
multi-vector-scene-mix-cgbbg-3-minio-2                            1/1     Running       0               16m     10.104.26.85    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-minio-3                            1/1     Running       0               16m     10.104.16.197   4am-node21   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-bookie-0                    1/1     Running       0               16m     10.104.33.35    4am-node36   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-bookie-1                    1/1     Running       0               16m     10.104.26.82    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-bookie-2                    1/1     Running       0               16m     10.104.15.172   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-bookie-init-26kmw           0/1     Completed     0               16m     10.104.15.110   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-broker-0                    1/1     Running       0               16m     10.104.15.113   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-proxy-0                     1/1     Running       0               16m     10.104.15.111   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-pulsar-init-9pb49           0/1     Completed     0               16m     10.104.33.251   4am-node36   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-recovery-0                  1/1     Running       0               16m     10.104.15.112   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-zookeeper-0                 1/1     Running       0               16m     10.104.26.80    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-zookeeper-1                 1/1     Running       0               10m     10.104.21.194   4am-node24   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-zookeeper-2                 1/1     Running       0               5m10s   10.104.34.239   4am-node37   <none>           <none> (base.py:257)
[2024-01-29 06:57:25,337 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'STATUS|multi-vector-scene-mix-cgbbg-3-milvus|multi-vector-scene-mix-cgbbg-3-minio|multi-vector-scene-mix-cgbbg-3-etcd|multi-vector-scene-mix-cgbbg-3-pulsar|multi-vector-scene-mix-cgbbg-3-kafka'  (util_cmd.py:14)
[2024-01-29 06:57:34,378 -  INFO - fouram]: [CliClient] pod details of release(multi-vector-scene-mix-cgbbg-3): 
 I0129 06:57:26.587740     537 request.go:665] Waited for 1.174724227s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/policy/v1beta1?timeout=32s
NAME                                                              READY   STATUS        RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-scene-mix-cgbbg-3-etcd-0                             1/1     Running       0               3h18m   10.104.33.60    4am-node36   <none>           <none>
multi-vector-scene-mix-cgbbg-3-etcd-1                             1/1     Running       2 (3h11m ago)   3h18m   10.104.15.154   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-etcd-2                             1/1     Running       0               3h18m   10.104.26.86    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-datacoord-746f8f464dc6r65   1/1     Running       2 (3h9m ago)    3h18m   10.104.26.24    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-datanode-c9fd959f9-4p4gc    1/1     Running       3 (3h5m ago)    3h18m   10.104.29.159   4am-node35   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-indexcoord-f56df7db7mj777   1/1     Running       0               3h18m   10.104.15.114   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-indexnode-5898675f6-bmpql   1/1     Running       1 (3h13m ago)   3h18m   10.104.26.25    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-proxy-dd77f56f9-7m68p       1/1     Running       3 (3h4m ago)    3h18m   10.104.26.27    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-querycoord-564478f47flq4c   1/1     Running       3 (3h5m ago)    3h18m   10.104.26.28    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-querynode-6bb7ff6c56bp2x5   1/1     Running       2 (3h9m ago)    3h18m   10.104.33.250   4am-node36   <none>           <none>
multi-vector-scene-mix-cgbbg-3-milvus-rootcoord-6f6d9f55fbxpvkt   1/1     Running       3 (3h7m ago)    3h18m   10.104.26.26    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-minio-0                            1/1     Running       0               3h18m   10.104.15.158   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-minio-1                            1/1     Running       0               3h18m   10.104.33.59    4am-node36   <none>           <none>
multi-vector-scene-mix-cgbbg-3-minio-2                            1/1     Running       0               3h18m   10.104.26.85    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-minio-3                            1/1     Running       0               3h18m   10.104.16.197   4am-node21   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-bookie-0                    1/1     Running       0               3h18m   10.104.33.35    4am-node36   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-bookie-1                    1/1     Running       0               3h18m   10.104.26.82    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-bookie-2                    1/1     Running       0               3h18m   10.104.15.172   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-bookie-init-26kmw           0/1     Completed     0               3h18m   10.104.15.110   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-broker-0                    1/1     Running       0               3h18m   10.104.15.113   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-proxy-0                     1/1     Running       0               3h18m   10.104.15.111   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-pulsar-init-9pb49           0/1     Completed     0               3h18m   10.104.33.251   4am-node36   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-recovery-0                  1/1     Running       0               3h18m   10.104.15.112   4am-node20   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-zookeeper-0                 1/1     Running       0               3h18m   10.104.26.80    4am-node32   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-zookeeper-1                 1/1     Running       0               3h12m   10.104.21.194   4am-node24   <none>           <none>
multi-vector-scene-mix-cgbbg-3-pulsar-zookeeper-2                 1/1     Running       0               3h7m    10.104.34.239   4am-node37   <none>           <none>

client pod name: multi-vector-scene-mix-cgbbg-2875428730
client log:
client.log

截屏2024-01-30 10 51 13 截屏2024-01-30 10 52 35

test result:

{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c8m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '32Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '17Gi'}},
                                     'replicas': 1},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '8Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '5Gi'}},
                                     'replicas': 1},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0',
                                                             'memory': '8Gi'},
                                                  'requests': {'cpu': '2.0',
                                                               'memory': '5Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': 'master-20240129-927d3109-amd64'}}},
            'host': 'multi-vector-scene-mix-cgbbg-3-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_dml_load_release_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {},
                                                                      'int64_1': {'index_type': 'INVERTED'},
                                                                      'varchar_1': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'DISKANN',
                                                                                         'index_param': {},
                                                                                         'metric_type': 'IP'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 2048},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_2': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_3': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 100000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'float_vector_2',
                                                                        'float_vector_3',
                                                                        'int64_1',
                                                                        'varchar_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT',
                                                  'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '3h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 8,
                                                       'params': {'nb': 1000,
                                                                  'timeout': 30,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 100000}},
                                                      {'type': 'delete',
                                                       'weight': 8,
                                                       'params': {'expr': '',
                                                                  'delete_length': 1000,
                                                                  'timeout': 30}},
                                                      {'type': 'flush',
                                                       'weight': 4,
                                                       'params': {'timeout': 180}},
                                                      {'type': 'load_release',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 600}}]},
            'run_id': 2024012995689735,
            'datetime': '2024-01-29 03:39:28.952123',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 31.7309,
                                      'float_vector_1': {'RT': 2.0246},
                                      'float_vector_2': {'RT': 11.1106},
                                      'float_vector_3': {'RT': 7.1028},
                                      'id': {'RT': 1.0244},
                                      'int64_1': {'RT': 1.0147},
                                      'varchar_1': {'RT': 1.0185}},
                            'insert': {'total_time': 11.0732,
                                       'VPS': 9030.8131,
                                       'batch_time': 1.1073,
                                       'batch': 10000},
                            'flush': {'RT': 3.0399},
                            'load': {'RT': 5.0392},
                            'Locust': {'Aggregated': {'Requests': 25839,
                                                      'Fails': 6,
                                                      'RPS': 2.41,
                                                      'fail_s': 0.0,
                                                      'RT_max': 438065.41,
                                                      'RT_avg': 8201.16,
                                                      'TP50': 550.0,
                                                      'TP99': 249000.0},
                                       'delete': {'Requests': 9749,
                                                  'Fails': 0,
                                                  'RPS': 0.91,
                                                  'fail_s': 0.0,
                                                  'RT_max': 2773.23,
                                                  'RT_avg': 430.56,
                                                  'TP50': 340.0,
                                                  'TP99': 1800.0},
                                       'flush': {'Requests': 4894,
                                                 'Fails': 6,
                                                 'RPS': 0.46,
                                                 'fail_s': 0.0,
                                                 'RT_max': 180455.43,
                                                 'RT_avg': 8472.0,
                                                 'TP50': 5300.0,
                                                 'TP99': 60000.0},
                                       'insert': {'Requests': 9982,
                                                  'Fails': 0,
                                                  'RPS': 0.93,
                                                  'fail_s': 0.0,
                                                  'RT_max': 5711.23,
                                                  'RT_avg': 537.64,
                                                  'TP50': 470.0,
                                                  'TP99': 1800.0},
                                       'load_release': {'Requests': 1214,
                                                        'Fails': 0,
                                                        'RPS': 0.11,
                                                        'fail_s': 0.0,
                                                        'RT_max': 438065.41,
                                                        'RT_avg': 132523.45,
                                                        'TP50': 86000.0,
                                                        'TP99': 419000.0}}}}}

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `DML & load -> release collection`
            verify DML & load -> release collection scenario,
            which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 128dim,
                'float_vector_3': 128dim,
                scalar field: int64_1, varchar_1
            2. build indexes:
                IVF_FLAT: 'float_vector'
                HNSW: 'float_vector_1',
                DISKANN: 'float_vector_2'
                IVF_SQ8: 'float_vector_3'
                INVERTED: 'int64_1', 'varchar_1'
                default scalar index: 'id'
            3. insert 100k data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 1
            7. concurrent request:
                - insert
                - delete
                - flush
                - load_release

Milvus Log

No response

Anything else?

No response
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test 2.4-features labels Jan 30, 2024
@wangting0128 wangting0128 added this to the 2.4.0 milestone Jan 30, 2024
@yanliang567
Copy link
Contributor

/assign @czs007
/unassign

@sre-ci-robot sre-ci-robot assigned czs007 and unassigned yanliang567 Jan 30, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 30, 2024
@czs007 czs007 assigned zhagnlu and unassigned czs007 Jan 30, 2024
@czs007
Copy link
Collaborator

czs007 commented Jan 30, 2024

@zhagnlu please help this

@zhagnlu
Copy link
Contributor

zhagnlu commented Jan 30, 2024 •
edited
Loading

Could you provide python client interface for test_hybrid_search_locust_dml_load_release_cluster, not familiar with hybrid search interface

@wangting0128
Copy link
Contributor Author

test_hybrid_search_locust_dml_load_release_cluster

@zhagnlu https://github.com/wangting0128/fouram/blob/7f01418f1f903f8e7120063223b4ad7e7e3fdbb0/testcases/benchmark/fearure.py#L2105

@zhagnlu
Copy link
Contributor

zhagnlu commented Jan 31, 2024

image
load segment for field id: 100 using field data
image
when next load segment, load index for field id:100, but this is index not field data, and load index check failed

@zhagnlu zhagnlu mentioned this issue Feb 1, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Feb 7, 2024
@zhagnlu
fix: erase pk empty check when pk index replace raw data (#30432)
e8a6f1e 
#30350

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
@wangting0128
Copy link
Contributor Author

wangting0128 commented Feb 22, 2024 •
edited
Loading

Recurrent

argo task: multi-vector-scene-mix-4w88w
test case name: test_hybrid_search_locust_dml_load_release_cluster
image: master-20240221-77eb6def

server:

NAME                                                              READY   STATUS        RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-scene-mix-4w88w-3-etcd-0                             1/1     Running       0                 3h13m   10.104.18.195   4am-node25   <none>           <none>
multi-vector-scene-mix-4w88w-3-etcd-1                             1/1     Running       0                 3h13m   10.104.17.99    4am-node23   <none>           <none>
multi-vector-scene-mix-4w88w-3-etcd-2                             1/1     Running       0                 3h13m   10.104.28.158   4am-node33   <none>           <none>
multi-vector-scene-mix-4w88w-3-milvus-datacoord-574c684765dwpq5   1/1     Running       1 (3h8m ago)      3h13m   10.104.29.128   4am-node35   <none>           <none>
multi-vector-scene-mix-4w88w-3-milvus-datanode-76879d7cd-bs8w6    1/1     Running       2 (3h4m ago)      3h13m   10.104.14.52    4am-node18   <none>           <none>
multi-vector-scene-mix-4w88w-3-milvus-indexcoord-9b44cd8ddx9mj7   1/1     Running       0                 3h13m   10.104.29.139   4am-node35   <none>           <none>
multi-vector-scene-mix-4w88w-3-milvus-indexnode-5cfb4d6d49w2fjb   1/1     Running       1 (3h8m ago)      3h13m   10.104.24.224   4am-node29   <none>           <none>
multi-vector-scene-mix-4w88w-3-milvus-proxy-545cf484d8-7g7wq      1/1     Running       2 (3h4m ago)      3h13m   10.104.29.138   4am-node35   <none>           <none>
multi-vector-scene-mix-4w88w-3-milvus-querycoord-755f5547c4hdhb   1/1     Running       2 (3h4m ago)      3h13m   10.104.29.137   4am-node35   <none>           <none>
multi-vector-scene-mix-4w88w-3-milvus-querynode-5849d5cf-ltbtk    1/1     Running       1 (3h8m ago)      3h13m   10.104.6.71     4am-node13   <none>           <none>
multi-vector-scene-mix-4w88w-3-milvus-rootcoord-5b857ff8975l6ft   1/1     Running       2 (3h4m ago)      3h13m   10.104.29.126   4am-node35   <none>           <none>
multi-vector-scene-mix-4w88w-3-minio-0                            1/1     Running       0                 3h13m   10.104.32.133   4am-node39   <none>           <none>
multi-vector-scene-mix-4w88w-3-minio-1                            1/1     Running       0                 3h13m   10.104.17.100   4am-node23   <none>           <none>
multi-vector-scene-mix-4w88w-3-minio-2                            1/1     Running       0                 3h13m   10.104.25.50    4am-node30   <none>           <none>
multi-vector-scene-mix-4w88w-3-minio-3                            1/1     Running       0                 3h13m   10.104.28.160   4am-node33   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-bookie-0                    1/1     Running       0                 3h13m   10.104.25.46    4am-node30   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-bookie-1                    1/1     Running       0                 3h13m   10.104.27.177   4am-node31   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-bookie-2                    1/1     Running       0                 3h13m   10.104.32.134   4am-node39   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-bookie-init-b2rm7           0/1     Completed     0                 3h13m   10.104.1.188    4am-node10   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-broker-0                    1/1     Running       0                 3h13m   10.104.4.86     4am-node11   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-proxy-0                     1/1     Running       0                 3h13m   10.104.1.191    4am-node10   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-pulsar-init-xljvn           0/1     Completed     0                 3h13m   10.104.6.70     4am-node13   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-recovery-0                  1/1     Running       0                 3h13m   10.104.4.85     4am-node11   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-zookeeper-0                 1/1     Running       0                 3h13m   10.104.16.6     4am-node21   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-zookeeper-1                 1/1     Running       0                 3h8m    10.104.25.55    4am-node30   <none>           <none>
multi-vector-scene-mix-4w88w-3-pulsar-zookeeper-2                 1/1     Running       0                 3h7m    10.104.17.120   4am-node23   <none>           <none>

client pod name:multi-vector-scene-mix-4w88w-3863944035
client log:
截屏2024-02-22 10 39 05

test steps:

        concurrent test and calculation of RT and QPS

        :purpose:  `DML & load -> release collection`
            verify DML & load -> release collection scenario,
            which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 128dim,
                'float_vector_3': 128dim,
                scalar field: int64_1, varchar_1
            2. build indexes:
                IVF_FLAT: 'float_vector'
                HNSW: 'float_vector_1',
                DISKANN: 'float_vector_2'
                IVF_SQ8: 'float_vector_3'
                INVERTED: 'int64_1', 'varchar_1'
                default scalar index: 'id'
            3. insert 100k data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 1
            7. concurrent request:
                - insert
                - delete
                - flush
                - load_release

test result:

{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c8m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '32Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '17Gi'}},
                                     'replicas': 1},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '8Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '5Gi'}},
                                     'replicas': 1},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0',
                                                             'memory': '8Gi'},
                                                  'requests': {'cpu': '2.0',
                                                               'memory': '5Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': 'master-20240221-77eb6def'}}},
            'host': 'multi-vector-scene-mix-4w88w-3-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_dml_load_release_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {},
                                                                      'int64_1': {'index_type': 'INVERTED'},
                                                                      'varchar_1': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8,
                                                                                                         'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'DISKANN',
                                                                                         'index_param': {},
                                                                                         'metric_type': 'IP'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 2048},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_2': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}},
                                                                       'float_vector_3': {'params': {'dim': 128},
                                                                                          'other_params': {'dataset': 'sift',
                                                                                                           'dim': 128}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 100000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'float_vector_2',
                                                                        'float_vector_3',
                                                                        'int64_1',
                                                                        'varchar_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT',
                                                  'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '3h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 8,
                                                       'params': {'nb': 1000,
                                                                  'timeout': 30,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 100000}},
                                                      {'type': 'delete',
                                                       'weight': 8,
                                                       'params': {'expr': '',
                                                                  'delete_length': 1000,
                                                                  'timeout': 30}},
                                                      {'type': 'flush',
                                                       'weight': 4,
                                                       'params': {'timeout': 180}},
                                                      {'type': 'load_release',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 600}}]},
            'run_id': 2024022121565004,
            'datetime': '2024-02-21 07:55:56.967338',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 45.7695,
                                      'float_vector_1': {'RT': 3.0284},
                                      'float_vector_2': {'RT': 12.5827},
                                      'float_vector_3': {'RT': 11.1593},
                                      'id': {'RT': 1.0158},
                                      'int64_1': {'RT': 0.518},
                                      'varchar_1': {'RT': 0.5175}},
                            'insert': {'total_time': 11.6856,
                                       'VPS': 8557.5409,
                                       'batch_time': 1.1686,
                                       'batch': 10000},
                            'flush': {'RT': 2.5247},
                            'load': {'RT': 5.0342},
                            'Locust': {'Aggregated': {'Requests': 23091,
                                                      'Fails': 2,
                                                      'RPS': 2.22,
                                                      'fail_s': 0.0,
                                                      'RT_max': 466858.77,
                                                      'RT_avg': 8952.46,
                                                      'TP50': 600.0,
                                                      'TP99': 260000.0},
                                       'delete': {'Requests': 8830,
                                                  'Fails': 0,
                                                  'RPS': 0.85,
                                                  'fail_s': 0.0,
                                                  'RT_max': 3208.08,
                                                  'RT_avg': 343.14,
                                                  'TP50': 200.0,
                                                  'TP99': 1700.0},
                                       'flush': {'Requests': 4421,
                                                 'Fails': 2,
                                                 'RPS': 0.42,
                                                 'fail_s': 0.0,
                                                 'RT_max': 180448.06,
                                                 'RT_avg': 9722.95,
                                                 'TP50': 5800.0,
                                                 'TP99': 60000.0},
                                       'insert': {'Requests': 8738,
                                                  'Fails': 0,
                                                  'RPS': 0.84,
                                                  'fail_s': 0.0,
                                                  'RT_max': 10821.32,
                                                  'RT_avg': 782.63,
                                                  'TP50': 560.0,
                                                  'TP99': 3300.0},
                                       'load_release': {'Requests': 1102,
                                                        'Fails': 0,
                                                        'RPS': 0.11,
                                                        'fail_s': 0.0,
                                                        'RT_max': 466858.77,
                                                        'RT_avg': 139625.69,
                                                        'TP50': 101000.0,
                                                        'TP99': 429000.0}}}}}

@zhagnlu

@wangting0128
Copy link
Contributor Author

verification passed

image：master-20240304-882b50c5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhagnlu zhagnlu

Labels
2.4-features kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.0
Development

No branches or pull requests
4 participants
@zhagnlu @wangting0128 @czs007 @yanliang567