[Bug]: Search return empty results while querynode recovering

[XuanYang-cn]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Expected Behavior

1. milvus master cluster 1 QN, 1 DN (milvus run querynode &> querynode.log &)
2. prepare a collection with 20 sealed segments, dim=128, numRows=5,000,000; create index, load.
3. Loop unlimited: Search with nq=1, print latency
4. start the second QN milvus run querynode --alias=1 &> querynode1.log &
5. stop the first QN: milvus stop querynode

Observe:

1. tail -f querynode1.log | grep "segments distribution"
2. tail -f proxy | grep "WARN"
3. The search latency of step3

Current Behaviour:

1. Before querynode1 load all segments done(collection available), proxy will get empty results.

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[XuanYang-cn]

See also #20502

[xiaofan-luan]

so this is only a issue on master right?

[XuanYang-cn]

so this is only a issue on master right?

I just tested on 2.2.0 and it reproduced.

[XuanYang-cn]

/assign @yah01
Please help investigating.

[XuanYang-cn]

/unassign

[yah01]

The QueryCoord detects the offline event of QueryNode in 1min (etcd lease timeout), during the lag, QueryCoord can't pull the distribution of the QueryNode, and considers it's online. The solutions are:

1. Check TargetID in QueryNode Query/Search
2. QueryCoord shouldn't returns QueryNodes whose distribution have been not updated for a while (much less than 1min)

[XuanYang-cn]

/assign
/unassign @yah01

[XuanYang-cn]

Still reproducing in master 243d8cf

/assign @yah01
/unassign

[yah01]

not reproduced with 11e44 for me

[yah01]

/assign @XuanYang-cn

[XuanYang-cn]

verified at e131915