[Bug]: [benchmark] Probabilistic failure to insert data

[elstic]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4-20240614-5fc1370f-amd64
- Deployment mode(standalone or cluster): all 
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task : fouram-disk-stab-1718560800, fouram-memory-index-stab-1718564400

client error log:

[2024-06-16 18:15:08,177 -  INFO - fouram]: [Base] Clean all index done. (base.py:513)
[2024-06-16 18:15:08,178 -  INFO - fouram]: [Base] Start build index of DISKANN for field:float_vector collection:fouram_hqE2VKMO, params:{'index_type': 'DISKANN', 'metric_type': 'L2', 'params': {}}, kwargs:{} (base.py:467)
[2024-06-16 18:15:08,716 -  INFO - fouram]: [Time] Index run in 0.5381s (api_request.py:49)
[2024-06-16 18:15:08,716 -  INFO - fouram]: [CommonCases] RT of build index DISKANN: 0.5381s (common_cases.py:154)
[2024-06-16 18:15:08,716 -  INFO - fouram]: [CommonCases] Prepare index DISKANN done. (common_cases.py:156)
[2024-06-16 18:15:08,717 -  INFO - fouram]: [CommonCases] No scalar and vector fields need to be indexed. (common_cases.py:175)
[2024-06-16 18:15:08,719 -  INFO - fouram]: [Base] Index params of fouram_hqE2VKMO:[{'float_vector': {'index_type': 'DISKANN', 'metric_type': 'L2', 'params': {}}}] (base.py:486)
[2024-06-16 18:15:08,720 -  INFO - fouram]: [Base] Start inserting 100000 vectors to collection fouram_hqE2VKMO (base.py:382)
[2024-06-16 18:15:08,798 -  INFO - fouram]: [Base] Start inserting, ids: 0 - 49999, data size: 100,000 (base.py:362)
[2024-06-16 18:15:09,102 - ERROR - fouram]: Unexpected error: [batch_insert], The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()., <Time: {'RPC start': '2024-06-16 18:15:09.102672', 'Exception': '2024-06-16 18:15:09.102817'}> (decorators.py:152)
[2024-06-16 18:15:09,106 - ERROR - fouram]: (api_response) : [Collection.insert] <MilvusException: (code=1, message=Unexpected error, message=<The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().>)>, [requestId: 5d47b43c-2c0c-11ef-8608-6e41b81294be] (api_request.py:57)
[2024-06-16 18:15:09,107 - ERROR - fouram]: [CheckFunc] insert request check failed, response:<MilvusException: (code=1, message=Unexpected error, message=<The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().>)> (func_check.py:54)

test env 4am, qa-milvus ns:

fouram-disk-sta60800-1-33-5050-etcd-0                             1/1     Running            0                7m40s   10.104.30.86    4am-node38   <none>           <none>
fouram-disk-sta60800-1-33-5050-etcd-1                             1/1     Running            0                7m40s   10.104.16.180   4am-node21   <none>           <none>
fouram-disk-sta60800-1-33-5050-etcd-2                             1/1     Running            0                7m40s   10.104.25.123   4am-node30   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-datacoord-6db598d876b6gxm   1/1     Running            4 (6m13s ago)    7m40s   10.104.15.35    4am-node20   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-datanode-5779f67657-fvxck   1/1     Running            4 (2m10s ago)    7m40s   10.104.26.197   4am-node32   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-indexcoord-98b65f7c-2j48r   1/1     Running            0                7m40s   10.104.24.122   4am-node29   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-indexnode-7df4555b75qt8px   1/1     Running            3 (6m37s ago)    7m40s   10.104.24.117   4am-node29   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-proxy-6678867859-4qpz9      1/1     Running            4 (2m10s ago)    7m40s   10.104.26.196   4am-node32   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-querycoord-78b747b8dfgd95   1/1     Running            4 (2m9s ago)     7m40s   10.104.27.78    4am-node31   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-querynode-6db5f49dd9v2jk7   1/1     Running            3 (6m44s ago)    7m40s   10.104.27.80    4am-node31   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-rootcoord-6dfd78cbc6dbr2f   1/1     Running            4 (6m17s ago)    7m40s   10.104.21.151   4am-node24   <none>           <none>
fouram-disk-sta60800-1-33-5050-minio-0                            1/1     Running            0                7m40s   10.104.32.39    4am-node39   <none>           <none>
fouram-disk-sta60800-1-33-5050-minio-1                            1/1     Running            0                7m40s   10.104.15.40    4am-node20   <none>           <none>
fouram-disk-sta60800-1-33-5050-minio-2                            1/1     Running            0                7m40s   10.104.16.184   4am-node21   <none>           <none>
fouram-disk-sta60800-1-33-5050-minio-3                            1/1     Running            0                7m39s   10.104.30.95    4am-node38   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-bookie-0                    1/1     Running            0                7m40s   10.104.25.121   4am-node30   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-bookie-1                    1/1     Running            0                7m40s   10.104.15.41    4am-node20   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-bookie-2                    1/1     Running            0                7m39s   10.104.16.185   4am-node21   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-bookie-init-9xt72           0/1     Completed          0                7m40s   10.104.16.176   4am-node21   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-broker-0                    1/1     Running            0                7m40s   10.104.21.153   4am-node24   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-proxy-0                     1/1     Running            0                7m40s   10.104.16.177   4am-node21   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-pulsar-init-s5bzb           0/1     Completed          0                7m40s   10.104.27.79    4am-node31   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-recovery-0                  1/1     Running            0                7m40s   10.104.5.33     4am-node12   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-zookeeper-0                 1/1     Running            0                7m40s   10.104.27.82    4am-node31   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-zookeeper-1                 1/1     Running            0                6m58s   10.104.23.19    4am-node27   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-zookeeper-2                 1/1     Running            0                6m2s    10.104.20.196   4am-node22   <none>           <none> (base.py:258)
[2024-06-16 18:15:09,529 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'NAME|fouram-disk-sta60800-1-33-5050-milvus|fouram-disk-sta60800-1-33-5050-minio|fouram-disk-sta60800-1-33-5050-etcd|fouram-disk-sta60800-1-33-5050-pulsar|fouram-disk-sta60800-1-33-5050-zookeeper|fouram-disk-sta60800-1-33-5050-kafka|fouram-disk-sta60800-1-33-5050-log|fouram-disk-sta60800-1-33-5050-tikv'  (util_cmd.py:14)
[2024-06-16 18:15:26,033 -  INFO - fouram]: [CliClient] pod details of release(fouram-disk-sta60800-1-33-5050): 
 I0616 18:15:10.799167    4090 request.go:665] Waited for 1.171512234s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/policy/v1?timeout=32s
I0616 18:15:20.799770    4090 request.go:665] Waited for 6.798195302s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/authentication.k8s.io/v1?timeout=32s
NAME                                                              READY   STATUS             RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouram-disk-sta60800-1-33-5050-etcd-0                             1/1     Running            0                7m58s   10.104.30.86    4am-node38   <none>           <none>
fouram-disk-sta60800-1-33-5050-etcd-1                             1/1     Running            0                7m58s   10.104.16.180   4am-node21   <none>           <none>
fouram-disk-sta60800-1-33-5050-etcd-2                             1/1     Running            0                7m58s   10.104.25.123   4am-node30   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-datacoord-6db598d876b6gxm   1/1     Running            4 (6m31s ago)    7m58s   10.104.15.35    4am-node20   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-datanode-5779f67657-fvxck   1/1     Running            4 (2m28s ago)    7m58s   10.104.26.197   4am-node32   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-indexcoord-98b65f7c-2j48r   1/1     Running            0                7m58s   10.104.24.122   4am-node29   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-indexnode-7df4555b75qt8px   1/1     Running            3 (6m55s ago)    7m58s   10.104.24.117   4am-node29   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-proxy-6678867859-4qpz9      1/1     Running            4 (2m28s ago)    7m58s   10.104.26.196   4am-node32   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-querycoord-78b747b8dfgd95   1/1     Running            4 (2m27s ago)    7m58s   10.104.27.78    4am-node31   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-querynode-6db5f49dd9v2jk7   1/1     Running            3 (7m2s ago)     7m58s   10.104.27.80    4am-node31   <none>           <none>
fouram-disk-sta60800-1-33-5050-milvus-rootcoord-6dfd78cbc6dbr2f   1/1     Running            4 (6m35s ago)    7m58s   10.104.21.151   4am-node24   <none>           <none>
fouram-disk-sta60800-1-33-5050-minio-0                            1/1     Running            0                7m58s   10.104.32.39    4am-node39   <none>           <none>
fouram-disk-sta60800-1-33-5050-minio-1                            1/1     Running            0                7m58s   10.104.15.40    4am-node20   <none>           <none>
fouram-disk-sta60800-1-33-5050-minio-2                            1/1     Running            0                7m58s   10.104.16.184   4am-node21   <none>           <none>
fouram-disk-sta60800-1-33-5050-minio-3                            1/1     Running            0                7m57s   10.104.30.95    4am-node38   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-bookie-0                    1/1     Running            0                7m58s   10.104.25.121   4am-node30   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-bookie-1                    1/1     Running            0                7m58s   10.104.15.41    4am-node20   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-bookie-2                    1/1     Running            0                7m57s   10.104.16.185   4am-node21   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-bookie-init-9xt72           0/1     Completed          0                7m58s   10.104.16.176   4am-node21   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-broker-0                    1/1     Running            0                7m58s   10.104.21.153   4am-node24   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-proxy-0                     1/1     Running            0                7m58s   10.104.16.177   4am-node21   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-pulsar-init-s5bzb           0/1     Completed          0                7m58s   10.104.27.79    4am-node31   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-recovery-0                  1/1     Running            0                7m58s   10.104.5.33     4am-node12   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-zookeeper-0                 1/1     Running            0                7m58s   10.104.27.82    4am-node31   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-zookeeper-1                 1/1     Running            0                7m16s   10.104.23.19    4am-node27   <none>           <none>
fouram-disk-sta60800-1-33-5050-pulsar-zookeeper-2                 1/1     Running            0                6m20s   10.104.20.196   4am-node22   <none>           <none>

About half of the stability tests fail due to insertion issues

Expected Behavior

No response

Steps To Reproduce

1. deploy milvus 
2. create collection 
3. create index (hnsw/diskann)
4. insert data =》 fail

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

The truth value of a Series is ambiguous. U

this seems to be a test script issue, not a milvus issue.

The truth value of a Series is ambiguous.

@elstic

[elstic]

The truth value of a Series is ambiguous. U

this seems to be a test script issue, not a milvus issue.

The truth value of a Series is ambiguous.

@elstic

It's been verified that it's not a milvus problem, it's an occasional bug in pymilvus that has been fixed.
We at nighlty were using an older version of pymilvus, which has now been updated to the latest version.