[Bug]: flush failed with error "channel not found"

[manishkuri]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.3.19
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): kafka
- SDK version(e.g. pymilvus v2.0.0rc2): python 2.3.3
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: N/A
- GPU: 
- Others:

Current Behavior

Inserts are getting completed successfully. But when doing flush we are getting DeadlineExceeded error in some cases. (success rate is 80%)
Flush [status: Failed] [code: 500] [sdk: Python-2.3.3] [msg: attempt #4: channel not found[channel=milvusdefault-rootcoord-dml_12_456467274406872959v2]] [traceID: ce0dc03593b636b45037c649e1e5ffa1]

Expected Behavior

Flush availablity should not drop to 80%

Steps To Reproduce

We checked the health of the MSK and etcd, everything looks fine.
We are attaching one of the traceID where it failed 
TraceId : 5f3a1b501e2c7ec724ae46e4f5b9497b

Attaching traceId logs, data coord , data worker, query worker logs for t-15 to t+15 of this trace Id
gdrive link for logs 

Milvus Log

gdrive link for logs :
https://drive.google.com/drive/folders/1WZEKgkYNglQRPX8t8FB-ZXza163YKtvn?usp=sharing

Anything else?

No response

[xiaofan-luan]

@weiliu1031
could you help on invesigating this.

This is an old cluster, just check if there some existing issue for this

[yanliang567]

/assign @weiliu1031
/unassign

[weiliu1031]

@manishkuri gdrive link for logs is empty, can you help to verify it?