Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][cluster] queryNode keeps restarting in DML & DQL scene #34508

Closed
1 task done
wangting0128 opened this issue Jul 9, 2024 · 2 comments
Closed
1 task done

[Bug]: [benchmark][cluster] queryNode keeps restarting in DML & DQL scene #34508

wangting0128 opened this issue Jul 9, 2024 · 2 comments
Assignees
@congqixia
Labels
kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.6

Comments

@wangting0128
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:2.4-20240708-f6cd8416-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka): pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: inverted-corn-1720472400
test case name: test_inverted_locust_hnsw_ivf_sq8_dml_dql_cluster

server:

[2024-07-09 00:52:02,891 -  INFO - fouram]: [Base] Deploy initial state: 
I0708 21:12:35.665616     379 request.go:665] Waited for 1.168715573s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/admissionregistration.k8s.io/v1?timeout=32s
I0708 21:12:45.864774     379 request.go:665] Waited for 4.197621032s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/pingcap.com/v1alpha1?timeout=32s
NAME                                                              READY   STATUS             RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
inverted-corn-172400-7-58-2776-etcd-0                             1/1     Running            0               8m42s   10.104.32.213   4am-node39   <none>           <none>
inverted-corn-172400-7-58-2776-etcd-1                             1/1     Running            0               8m42s   10.104.26.221   4am-node32   <none>           <none>
inverted-corn-172400-7-58-2776-etcd-2                             1/1     Running            0               8m42s   10.104.27.84    4am-node31   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-datacoord-bc45b477f-5dbmg   1/1     Running            4 (7m18s ago)   8m42s   10.104.14.249   4am-node18   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-datanode-dd96ffb68-nb6m5    1/1     Running            5 (2m12s ago)   8m42s   10.104.14.250   4am-node18   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-indexcoord-77b7cd4f9qq797   1/1     Running            0               8m42s   10.104.14.248   4am-node18   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-indexnode-64f599cfdc28mvd   1/1     Running            4 (7m4s ago)    8m42s   10.104.9.241    4am-node14   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-indexnode-64f599cfdcq6k42   1/1     Running            4 (7m7s ago)    8m42s   10.104.6.161    4am-node13   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-proxy-6df89d648b-tbvpp      1/1     Running            5 (2m12s ago)   8m42s   10.104.13.38    4am-node16   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-querycoord-58c5658bfcmtw7   1/1     Running            5 (2m12s ago)   8m42s   10.104.18.209   4am-node25   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-querynode-7469bbd449fxxw5   1/1     Running            4 (7m5s ago)    8m42s   10.104.18.210   4am-node25   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-rootcoord-666b97d56cz8lkk   1/1     Running            5 (2m12s ago)   8m42s   10.104.14.251   4am-node18   <none>           <none>
inverted-corn-172400-7-58-2776-minio-0                            1/1     Running            0               8m42s   10.104.26.222   4am-node32   <none>           <none>
inverted-corn-172400-7-58-2776-minio-1                            1/1     Running            0               8m42s   10.104.24.77    4am-node29   <none>           <none>
inverted-corn-172400-7-58-2776-minio-2                            1/1     Running            0               8m42s   10.104.34.62    4am-node37   <none>           <none>
inverted-corn-172400-7-58-2776-minio-3                            1/1     Running            0               8m41s   10.104.32.217   4am-node39   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-bookie-0                    1/1     Running            0               8m42s   10.104.27.82    4am-node31   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-bookie-1                    1/1     Running            0               8m42s   10.104.32.216   4am-node39   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-bookie-2                    1/1     Running            0               8m42s   10.104.19.62    4am-node28   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-bookie-init-lcjtk           0/1     Completed          0               8m42s   10.104.13.36    4am-node16   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-broker-0                    1/1     Running            0               8m42s   10.104.19.45    4am-node28   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-proxy-0                     1/1     Running            0               8m42s   10.104.6.160    4am-node13   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-pulsar-init-rxczd           0/1     Completed          0               8m42s   10.104.19.44    4am-node28   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-recovery-0                  1/1     Running            0               8m42s   10.104.15.137   4am-node20   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-zookeeper-0                 1/1     Running            0               8m42s   10.104.15.149   4am-node20   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-zookeeper-1                 1/1     Running            0               7m14s   10.104.24.104   4am-node29   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-zookeeper-2                 1/1     Running            0               5m17s   10.104.32.236   4am-node39   <none>           <none> (base.py:261)
[2024-07-09 00:52:02,892 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'NAME|inverted-corn-172400-7-58-2776-milvus|inverted-corn-172400-7-58-2776-minio|inverted-corn-172400-7-58-2776-etcd|inverted-corn-172400-7-58-2776-pulsar|inverted-corn-172400-7-58-2776-zookeeper|inverted-corn-172400-7-58-2776-kafka|inverted-corn-172400-7-58-2776-log|inverted-corn-172400-7-58-2776-tikv'  (util_cmd.py:14)
[2024-07-09 00:52:23,342 -  INFO - fouram]: [CliClient] pod details of release(inverted-corn-172400-7-58-2776): 
 I0709 00:52:04.136158     486 request.go:665] Waited for 1.174260238s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/notification.kubesphere.io/v2beta1?timeout=32s
I0709 00:52:14.136213     486 request.go:665] Waited for 3.995388082s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/apiextensions.k8s.io/v1?timeout=32s
NAME                                                              READY   STATUS              RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
inverted-corn-172400-7-58-2776-etcd-0                             1/1     Running             0                3h48m   10.104.32.213   4am-node39   <none>           <none>
inverted-corn-172400-7-58-2776-etcd-1                             1/1     Running             0                3h48m   10.104.26.221   4am-node32   <none>           <none>
inverted-corn-172400-7-58-2776-etcd-2                             1/1     Running             0                3h48m   10.104.27.84    4am-node31   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-datacoord-bc45b477f-5dbmg   1/1     Running             4 (3h46m ago)    3h48m   10.104.14.249   4am-node18   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-datanode-dd96ffb68-nb6m5    1/1     Running             5 (3h41m ago)    3h48m   10.104.14.250   4am-node18   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-indexcoord-77b7cd4f9qq797   1/1     Running             0                3h48m   10.104.14.248   4am-node18   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-indexnode-64f599cfdc28mvd   1/1     Running             4 (3h46m ago)    3h48m   10.104.9.241    4am-node14   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-indexnode-64f599cfdcq6k42   1/1     Running             4 (3h46m ago)    3h48m   10.104.6.161    4am-node13   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-proxy-6df89d648b-tbvpp      1/1     Running             5 (3h41m ago)    3h48m   10.104.13.38    4am-node16   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-querycoord-58c5658bfcmtw7   1/1     Running             5 (3h41m ago)    3h48m   10.104.18.209   4am-node25   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-querynode-7469bbd449fxxw5   0/1     Running             40 (5m40s ago)   3h48m   10.104.18.210   4am-node25   <none>           <none>
inverted-corn-172400-7-58-2776-milvus-rootcoord-666b97d56cz8lkk   1/1     Running             5 (3h41m ago)    3h48m   10.104.14.251   4am-node18   <none>           <none>
inverted-corn-172400-7-58-2776-minio-0                            1/1     Running             0                3h48m   10.104.26.222   4am-node32   <none>           <none>
inverted-corn-172400-7-58-2776-minio-1                            1/1     Running             0                3h48m   10.104.24.77    4am-node29   <none>           <none>
inverted-corn-172400-7-58-2776-minio-2                            1/1     Running             0                3h48m   10.104.34.62    4am-node37   <none>           <none>
inverted-corn-172400-7-58-2776-minio-3                            1/1     Running             0                3h48m   10.104.32.217   4am-node39   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-bookie-0                    1/1     Running             0                3h48m   10.104.27.82    4am-node31   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-bookie-1                    1/1     Running             0                3h48m   10.104.32.216   4am-node39   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-bookie-2                    1/1     Running             0                3h48m   10.104.19.62    4am-node28   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-bookie-init-lcjtk           0/1     Completed           0                3h48m   10.104.13.36    4am-node16   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-broker-0                    1/1     Running             0                3h48m   10.104.19.45    4am-node28   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-proxy-0                     1/1     Running             0                3h48m   10.104.6.160    4am-node13   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-pulsar-init-rxczd           0/1     Completed           0                3h48m   10.104.19.44    4am-node28   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-recovery-0                  1/1     Running             0                3h48m   10.104.15.137   4am-node20   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-zookeeper-0                 1/1     Running             0                3h48m   10.104.15.149   4am-node20   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-zookeeper-1                 1/1     Running             0                3h46m   10.104.24.104   4am-node29   <none>           <none>
inverted-corn-172400-7-58-2776-pulsar-zookeeper-2                 1/1     Running             0                3h44m   10.104.32.236   4am-node39   <none>           <none> (cli_client.py:144)

inverted-corn-172400-7-58-2776-milvus-querynode-7469bbd449fxxw5.log

截屏2024-07-09 11 08 49

client pod name: inverted-corn-1720472400-3806193430

Expected Behavior

No response

Steps To Reproduce

[2024-07-08 21:04:09,421 -  INFO - fouram]: [setup_method] Start setup test case test_inverted_locust_hnsw_ivf_sq8_dml_dql_cluster, test document:
        concurrent test and calculation of RT and QPS

        :purpose:  `vector: memory index`
            verify concurrent DML & DQL scenario which has 2 float_vector fields & 16 scalar fields

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 200dim,
                'int8_1', 'int16_1', 'int32_1', 'int64_1', 'double_1', 'float_1', 'varchar_1', 'bool_1',
                'int8_2', 'int16_2', 'int32_2', 'int64_2', 'double_2', 'float_2', 'varchar_2', 'bool_2'
            2. build indexes:
                HNSW: 'float_vector'
                IVF_SQ8: 'float_vector_1'
                scalar_default_index: 'int8_1', 'int16_1', 'int32_1', 'int64_1', 'double_1', 'float_1', 'varchar_1'
                scalar_INVERTED_index: 'int8_2', 'int16_2', 'int32_2', 'int64_2', 'double_2', 'float_2', 'varchar_2', 'bool_2'
            3. insert 5 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
            7. concurrent request:
                - insert
                - delete
                - flush
                - load
                - search
                - hybrid_search
                - query

Milvus Log

No response

Anything else?

test result:

[2024-07-09 00:51:38,549 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-07-09 00:51:38,550 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-07-09 00:51:38,550 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-07-09 00:51:38,550 -  INFO - fouram]: grpc     delete                                                                          3351     0(0.00%) |     12       4     409      8 |    0.31        0.00 (stats.py:789)
[2024-07-09 00:51:38,550 -  INFO - fouram]: grpc     flush                                                                           3457    41(1.19%) |  60259     242  602093   3200 |    0.32        0.00 (stats.py:789)
[2024-07-09 00:51:38,550 -  INFO - fouram]: grpc     hybrid_search                                                                   3406 3212(94.30%) |    218       2   54032      4 |    0.32        0.30 (stats.py:789)
[2024-07-09 00:51:38,550 -  INFO - fouram]: grpc     insert                                                                          3550     0(0.00%) |     23       8     805     15 |    0.33        0.00 (stats.py:789)
[2024-07-09 00:51:38,550 -  INFO - fouram]: grpc     load                                                                            3494     0(0.00%) |     61       8     989     56 |    0.32        0.00 (stats.py:789)
[2024-07-09 00:51:38,550 -  INFO - fouram]: grpc     query                                                                           3487 3322(95.27%) |    554       1   55396      3 |    0.32        0.31 (stats.py:789)
[2024-07-09 00:51:38,550 -  INFO - fouram]: grpc     search                                                                          3358 3174(94.52%) |    508      80   53873     87 |    0.31        0.29 (stats.py:789)
[2024-07-09 00:51:38,550 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-07-09 00:51:38,551 -  INFO - fouram]:          Aggregated                                                                     24103 9749(40.45%) |   8838       1  602093     17 |    2.23        0.90 (stats.py:789)
[2024-07-09 00:51:38,551 -  INFO - fouram]:  (stats.py:790)
[2024-07-09 00:51:38,554 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_8c16m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '16.0',
                                                              'memory': '64Gi'},
                                                   'requests': {'cpu': '9.0',
                                                                'memory': '33Gi'}},
                                     'replicas': 1},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '16Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '9Gi'}},
                                     'replicas': 2},
                       'dataNode': {'resources': {'limits': {'cpu': '8.0',
                                                             'memory': '16Gi'},
                                                  'requests': {'cpu': '5.0',
                                                               'memory': '9Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': '2.4-20240708-f6cd8416-amd64'}}},
            'host': 'inverted-corn-172400-7-58-2776-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_inverted_locust_hnsw_ivf_sq8_dml_dql_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'int8_1': {},
                                                                      'int16_1': {},
                                                                      'int32_1': {},
                                                                      'int64_1': {},
                                                                      'double_1': {},
                                                                      'float_1': {},
                                                                      'varchar_1': {},
                                                                      'int8_2': {'index_type': 'INVERTED'},
                                                                      'int16_2': {'index_type': 'INVERTED'},
                                                                      'int32_2': {'index_type': 'INVERTED'},
                                                                      'int64_2': {'index_type': 'INVERTED'},
                                                                      'double_2': {'index_type': 'INVERTED'},
                                                                      'float_2': {'index_type': 'INVERTED'},
                                                                      'varchar_2': {'index_type': 'INVERTED'},
                                                                      'bool_2': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 1024},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 200},
                                                                                          'other_params': {'dataset': 'text2img'}}},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 5000000,
                                                    'ni_per': 5000},
                                 'collection_params': {'other_fields': ['float_vector_1',
                                                                        'int8_1',
                                                                        'int16_1',
                                                                        'int32_1',
                                                                        'int64_1',
                                                                        'double_1',
                                                                        'float_1',
                                                                        'varchar_1',
                                                                        'bool_1',
                                                                        'int8_2',
                                                                        'int16_2',
                                                                        'int32_2',
                                                                        'int64_2',
                                                                        'double_2',
                                                                        'float_2',
                                                                        'varchar_2',
                                                                        'bool_2'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 8,
                                                                  'efConstruction': 200}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '3h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 10,
                                                                  'timeout': 30,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 5000000}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'expr': '',
                                                                  'delete_length': 9,
                                                                  'timeout': 30}},
                                                      {'type': 'flush',
                                                       'weight': 1,
                                                       'params': {'timeout': 600}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 180}},
                                                      {'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'ef': 64},
                                                                  'expr': 'int64_1 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&& '
                                                                          'id '
                                                                          '> '
                                                                          '-1',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 180,
                                                                  'random_data': True}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 10,
                                                                  'reqs': [{'search_param': {'nprobe': 16},
                                                                            'anns_field': 'float_vector_1',
                                                                            'expr': 'varchar_1 '
                                                                                    'like '
                                                                                    '"0%" '
                                                                                    '&& '
                                                                                    'bool_2 '
                                                                                    '== '
                                                                                    'True',
                                                                            'top_k': 2000},
                                                                           {'search_param': {'ef': 128},
                                                                            'anns_field': 'float_vector',
                                                                            'expr': 'int64_1 '
                                                                                    '< '
                                                                                    '100000 '
                                                                                    '&& '
                                                                                    'float_2 '
                                                                                    '> '
                                                                                    '10.0'}],
                                                                  'rerank': {'WeightedRanker': [0.5,
                                                                                                0.5]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'timeout': 60,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&&  '
                                                                          'int64_2 '
                                                                          '> '
                                                                          '-1 '
                                                                          '&& ',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 180,
                                                                  'random_data': True,
                                                                  'random_count': 20,
                                                                  'random_range': [2500000.0,
                                                                                   5000000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64'}}]},
            'run_id': 2024070826492400,
            'datetime': '2024-07-08 21:04:09.421598',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 779.8129,
                                      'float_vector_1': {'RT': 434.1661},
                                      'int8_1': {'RT': 160.1804},
                                      'int16_1': {'RT': 117.8104},
                                      'int32_1': {'RT': 99.2793},
                                      'int64_1': {'RT': 0.545},
                                      'double_1': {'RT': 0.5884},
                                      'float_1': {'RT': 0.5387},
                                      'varchar_1': {'RT': 0.5278},
                                      'int8_2': {'RT': 0.5164},
                                      'int16_2': {'RT': 18.1675},
                                      'int32_2': {'RT': 0.5192},
                                      'int64_2': {'RT': 0.5174},
                                      'double_2': {'RT': 0.5191},
                                      'float_2': {'RT': 0.5184},
                                      'varchar_2': {'RT': 0.5181},
                                      'bool_2': {'RT': 0.5205}},
                            'insert': {'total_time': 601.6901,
                                       'VPS': 8309.9257,
                                       'batch_time': 0.6017,
                                       'batch': 5000},
                            'flush': {'RT': 3.0481},
                            'load': {'RT': 9.0908},
                            'Locust': {'Aggregated': {'Requests': 24103,
                                                      'Fails': 9749,
                                                      'RPS': 2.23,
                                                      'fail_s': 0.4,
                                                      'RT_max': 602093.07,
                                                      'RT_avg': 8838.96,
                                                      'TP50': 17,
                                                      'TP99': 376000.0},
                                       'delete': {'Requests': 3351,
                                                  'Fails': 0,
                                                  'RPS': 0.31,
                                                  'fail_s': 0.0,
                                                  'RT_max': 409.59,
                                                  'RT_avg': 12.37,
                                                  'TP50': 8,
                                                  'TP99': 93},
                                       'flush': {'Requests': 3457,
                                                 'Fails': 41,
                                                 'RPS': 0.32,
                                                 'fail_s': 0.01,
                                                 'RT_max': 602093.07,
                                                 'RT_avg': 60259.69,
                                                 'TP50': 3200.0,
                                                 'TP99': 601000.0},
                                       'hybrid_search': {'Requests': 3406,
                                                         'Fails': 3212,
                                                         'RPS': 0.32,
                                                         'fail_s': 0.94,
                                                         'RT_max': 54032.59,
                                                         'RT_avg': 218.9,
                                                         'TP50': 4,
                                                         'TP99': 690.0},
                                       'insert': {'Requests': 3550,
                                                  'Fails': 0,
                                                  'RPS': 0.33,
                                                  'fail_s': 0.0,
                                                  'RT_max': 805.93,
                                                  'RT_avg': 23.94,
                                                  'TP50': 15,
                                                  'TP99': 180.0},
                                       'load': {'Requests': 3494,
                                                'Fails': 0,
                                                'RPS': 0.32,
                                                'fail_s': 0.0,
                                                'RT_max': 989.53,
                                                'RT_avg': 61.41,
                                                'TP50': 56,
                                                'TP99': 190.0},
                                       'query': {'Requests': 3487,
                                                 'Fails': 3322,
                                                 'RPS': 0.32,
                                                 'fail_s': 0.95,
                                                 'RT_max': 55396.88,
                                                 'RT_avg': 554.13,
                                                 'TP50': 3,
                                                 'TP99': 33000.0},
                                       'search': {'Requests': 3358,
                                                  'Fails': 3174,
                                                  'RPS': 0.31,
                                                  'fail_s': 0.95,
                                                  'RT_max': 53873.82,
                                                  'RT_avg': 508.9,
                                                  'TP50': 87,
                                                  'TP99': 13000.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Jul 9, 2024
@wangting0128 wangting0128 added this to the 2.4.6 milestone Jul 9, 2024
@yanliang567 yanliang567 added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 9, 2024
@yanliang567
Copy link
Contributor

/assign @congqixia
/unassign

@sre-ci-robot sre-ci-robot assigned congqixia and unassigned yanliang567 Jul 9, 2024
congqixia added a commit to congqixia/milvus that referenced this issue Jul 9, 2024
@congqixia
fix: Write padding at end of mmap file not chunk
c8d08d3 
Related to milvus-io#34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia congqixia mentioned this issue Jul 9, 2024
Merged
congqixia added a commit to congqixia/milvus that referenced this issue Jul 9, 2024
@congqixia
fix: Write padding at end of mmap file not chunk
f40309c 
Related to milvus-io#34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia congqixia mentioned this issue Jul 9, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Jul 9, 2024
@congqixia
fix: [2.4] Write padding at end of mmap file not chunk (#34530)
0d8defb 
Cherry-pick from master
pr: #34529
Related to #34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@wangting0128
Copy link
Contributor Author

verification passed

argo task: inverted-corn-1720558800
image: 2.4-20240709-0d8defb1-amd64

sre-ci-robot pushed a commit that referenced this issue Jul 10, 2024
@congqixia
fix: Write padding at end of mmap file not chunk (#34529)
4850336 
Related to #34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@congqixia congqixia

Labels
kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.6
Development

No branches or pull requests
3 participants
@wangting0128 @yanliang567 @congqixia