[Bug]: Add Tickle logic in channel reassign

[xiaofan-luan]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Current Data coord send a channel watch to datanode, and wait for Params.DataCoordCfg.MaxWatchDuration time before it decide to reassign the channel to another. datanode.

This may cause serious issues, especially when watch channel takes lager than Params.DataCoordCfg.MaxWatchDuration(happens on the case where there are many stats logs).

Expected Behavior

1. Datanode keep modifying etcd path(Tickle), so datacoord won't timeout the assign, after tickle datacoord will only reset timeout and continue.

2. Thinks of some way to improve stats log load. This becomes a serious issue when data channel becomes huge.

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[xiaofan-luan]

/assign @aoiasd

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[xiaofan-luan]

keep it open

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.