[Bug]: It's been a long time since Milvus recovered from the standalone pod kill chaos test, but the client still can't connect.

[zhuwenxing]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:master-20240531-2c7bb0b8-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2024-06-02T19:45:43.000Z] [2024-06-02 19:45:42 - WARNING - pymilvus.decorators]: [batch_insert] retry:36, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused> (decorators.py:99)

[2024-06-02T19:45:43.559Z] [2024-06-02 19:45:43 - WARNING - pymilvus.decorators]: [query] retry:30, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused> (decorators.py:99)

[2024-06-02T19:45:44.484Z] [2024-06-02 19:45:44 - WARNING - pymilvus.decorators]: [batch_insert] retry:22, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused> (decorators.py:99)

[2024-06-02T19:45:45.044Z] [2024-06-02 19:45:44 - WARNING - pymilvus.decorators]: [drop_collection] retry:14, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused> (decorators.py:99)

[2024-06-02T19:45:45.044Z] [2024-06-02 19:45:44 - WARNING - pymilvus.decorators]: [has_collection] retry:20, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused> (decorators.py:99)

[2024-06-02T19:45:45.967Z] [2024-06-02 19:45:45 - ERROR - pymilvus.decorators]: RPC error: [query], <MilvusException: (code=<bound method _MultiThreadedRendezvous.code of <_MultiThreadedRendezvous of RPC that terminated with:

[2024-06-02T19:45:45.967Z] 	status = StatusCode.UNAVAILABLE

[2024-06-02T19:45:45.967Z] 	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused"

[2024-06-02T19:45:45.967Z] 	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused {created_time:"2024-06-02T19:45:45.71082718+00:00", grpc_status:14}"

[2024-06-02T19:45:45.967Z] >>, message=Retry timeout: 10s, message=failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused)>, <Time:{'RPC start': '2024-06-02 19:45:35.625131', 'RPC error': '2024-06-02 19:45:45.711242'}> (decorators.py:139)

[2024-06-02T19:45:45.967Z] [2024-06-02 19:45:45 - ERROR - ci_test]: Traceback (most recent call last):

[2024-06-02T19:45:45.967Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper

[2024-06-02T19:45:45.967Z]     res = func(*args, **_kwargs)

[2024-06-02T19:45:45.967Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request

[2024-06-02T19:45:45.967Z]     return func(*arg, **kwargs)

[2024-06-02T19:45:45.967Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 1078, in query

[2024-06-02T19:45:45.967Z]     return conn.query(

[2024-06-02T19:45:45.967Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 140, in handler

[2024-06-02T19:45:45.967Z]     raise e from e

[2024-06-02T19:45:45.967Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler

[2024-06-02T19:45:45.967Z]     return func(*args, **kwargs)

[2024-06-02T19:45:45.967Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 175, in handler

[2024-06-02T19:45:45.967Z]     return func(self, *args, **kwargs)

[2024-06-02T19:45:45.967Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 92, in handler

[2024-06-02T19:45:45.967Z]     raise MilvusException(e.code, f"{to_msg}, message={e.details()}") from e

[2024-06-02T19:45:45.967Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=<bound method _MultiThreadedRendezvous.code of <_MultiThreadedRendezvous of RPC that terminated with:

[2024-06-02T19:45:45.967Z] 	status = StatusCode.UNAVAILABLE

[2024-06-02T19:45:45.968Z] 	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused"

[2024-06-02T19:45:45.968Z] 	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused {created_time:"2024-06-02T19:45:45.71082718+00:00", grpc_status:14}"

[2024-06-02T19:45:45.968Z] >>, message=Retry timeout: 10s, message=failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused)>

[2024-06-02T19:45:45.968Z]  (api_request.py:45)

[2024-06-02T19:45:45.968Z] [2024-06-02 19:45:45 - ERROR - ci_test]: (api_response) : <MilvusException: (code=<bound method _MultiThreadedRendezvous.code of <_MultiThreadedRendezvous of RPC that terminated with:

[2024-06-02T19:45:45.968Z] 	status = StatusCode.UNAVAILABLE

[2024-06-02T19:45:45.968Z] 	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection r...... (api_request.py:46)

[2024-06-02T19:45:46.528Z] [2024-06-02 19:45:46 - WARNING - pymilvus.decorators]: [batch_insert] retry:37, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.83.107:19530: Failed to connect to remote host: Connection refused> (decorators.py:99)

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-kafka-cron/detail/chaos-test-kafka-cron/14549/pipeline

log:
artifacts-standalone-pod-kill-14549-server-logs.tar.gz

Anything else?

No response

[yanliang567]

/assign @weiliu1031
/unassign