[Bug]: channel by-dev-rootcoord-dml_11_443279871761864895v0 is not available in any replica

[XDeviation]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.2.12
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.3.1
- OS(Ubuntu or CentOS): Ubuntu 20.04
- CPU/Memory: ...
- GPU: None
- Others:

Current Behavior

schema:
{'auto_id': True, 'description': 'Object search collection', 'fields': [{'name': 'pk', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 1280}}], 'enable_dynamic_field': True}
collection.num_entities: 1000000

RPC error: [query], <MilvusException: (code=1, message=fail to query on all shard leaders, err=All attempts results:
attempt #1:All attempts results:
attempt #1:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_12_443279871761864895v1 is not available in any replica, err=<nil>
attempt #2:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_12_443279871761864895v1 is not available in any replica, err=<nil>
attempt #3:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_11_443279871761864895v0 is not available in any replica, err=<nil>
attempt #4:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_11_443279871761864895v0 is not available in any replica, err=<nil>
attempt #5:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_11_443279871761864895v0 is not available in any replica, err=<nil>
attempt #6:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_11_443279871761864895v0 is not available in any replica, err=<nil>
attempt #7:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_11_443279871761864895v0 is not available in any replica, err=<nil>
attempt #8:context deadline exceeded

attempt #2:context canceled
)>, <Time:{'RPC start': '2023-09-25 05:17:59.793247', 'RPC error': '2023-09-25 05:18:09.799420'}>

Expected Behavior

all test cases passed

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[yanliang567]

@XDeviation please provide the full milvus logs for investigation. refer this doc to export the whole Milvus logs. For Milvus installed with docker-compose, you can use docker-compose logs > milvus.log to export the logs.
Also if possible, please share the code snippet to reproduce the issue.

/assign @XDeviation
/unassign

[axiangcoding]

Same issue.
In my case, I use langchain vectorstore integrate pymlivus to use Milvus. This error occurs with probability, but not frequently.

[xiaofan-luan]

Same issue. In my case, I use langchain vectorstore integrate pymlivus to use Milvus. This error occurs with probability, but not frequently.

could you offer logs for milvus? you can use export-log.sh

[axiangcoding]

Same issue. In my case, I use langchain vectorstore integrate pymlivus to use Milvus. This error occurs with probability, but not frequently.

could you offer logs for milvus? you can use export-log.sh

It's been a long time, I will export the log next time it happens.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[ronghuaihai]

we have face the same question，

the log of proxy as follows：
[2023/12/09 08:38:46.775 +00:00] [WARN] [retry/retry.go:44] ["retry func failed"] ["retry time"=0] [error="All attempts results:\nattempt #1:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #2:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #3:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #4:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #5:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #6:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #7:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #8:context deadline exceeded\n"]

how to solve the problem

[xiaofan-luan]

we have face the same question，

the log of proxy as follows： [2023/12/09 08:38:46.775 +00:00] [WARN] [retry/retry.go:44] ["retry func failed"] ["retry time"=0] [error="All attempts results:\nattempt #1:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #2:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #3:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #4:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #5:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #6:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #7:fail to get shard leaders from QueryCoord: channel by-dev-rootcoord-dml_4_443999969427532581v1 is not available in any replica, err=NodeOffline(nodeID=-1)\nattempt #8:context deadline exceeded\n"]

how to solve the problem

please offer full log and open a seperate issue, thanks.