[Bug]: Zero Recall Rate, Very Strange for multi-vector search!!!!!!!

[JackTan25]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: build from source commit 648d5661ca8771fadc664f427ac330b083b8734e
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus  2.4.3
- OS(Ubuntu or CentOS): Ubuntu20.04
- CPU/Memory: 12c 128G
- GPU: 
- Others:

Current Behavior

Get Zero Recall Rate.

can't find any correct result.

Expected Behavior

expect high recall rate.

Steps To Reproduce

1. modify the rescore code like below:

func (ws *weightedScorer) getActivateFunc() activateFunc {
	mUpper := strings.ToUpper(ws.getMetricType())
	isCosine := mUpper == strings.ToUpper(metric.COSINE)
	isIP := mUpper == strings.ToUpper(metric.IP)
	if isCosine {
		f := func(distance float32) float32 {
			return (1 + distance) * 0.5
		}
		return f
	}

	if isIP {
		f := func(distance float32) float32 {
			return 0.5 + float32(math.Atan(float64(distance)))/math.Pi
		}
		return f
	}

	f := func(distance float32) float32 {
                // just return distance, because mine metric way is so.
		return distance
	}
	return f
}

2. use my script like this:
$ cd small_data
$ python3 data_load_small_data.py
$ python3 milvus_small_multi_vector.py > milvus.txt
$ python3 ground_truth.py > ground_truth>txt

Milvus Log

No response

Anything else?

No response

[JackTan25]

small_data.zip
@czs007

1. modify source code here
   
2. unzip small_data.zip & cd small_data
3. python3 data_load_small_data.py
4. python3 milvus_small_multi_vector.py > milvus.txt
5. python3 ground_truth.py > ground_truth.txt
6. we get the result comparison, distances and ids are all totally wrong. We get zero recall rate!!!! (the result is sorted by the distance).
   

[yanliang567]

/assign @czs007
/unassign

[JackTan25]

I'm in hurry, so please help me, thanks very much. cc @czs007. Can you reproduce?

[czs007]

working on it. @JackTan25

[JackTan25]

working on it. @JackTan25

Thanks, can you reproduce it? cc @czs007

[czs007]

@JackTan25

milvus/internal/proxy/search_reduce_util.go

Line 478 in 5452376

big := func(i, j int) bool {

Please change the function here from "big" to "small". After applying the previous activation function, when performing ranking, it was sorted in descending order. Once you modify it to return the original L2 distance, it should be sorted in ascending order.

[JackTan25]

Ok, let me try it. Thanks.

[JackTan25]

hi, I test 1000 rows dataset, and it works expectedly, but when the dataset is very large to 100w, I can get only 2%. I want to upper the limit 16384, where should I modify for the source code? @czs007

[czs007]

@JackTan25 what do you mean by 2%? recall?

[JackTan25]

yes， I upper the top to 99w and I can get 64% now.

[czs007]

@JackTan25 Why does the value of TopK need to be so large?

[JackTan25]

I feel strange yet. I think you can test the small_data I give you, when the topk2 is low(50), the recall rate is still zero, when I lift it up to 1000(dataset is 1000), it can only get 94%. Multi Vector's recall is very low, but Single Vector recall is very high and quick. cc @czs007

[JackTan25]

The question is that, the query itself can make a big difference to the result. Different query can get different recall.

[xiaofan-luan]

I feel strange yet. I think you can test the small_data I give you, when the topk2 is low(50), the recall rate is still zero, when I lift it up to 1000(dataset is 1000), it can only get 94%. Multi Vector's recall is very low, but Single Vector recall is very high and quick. cc @czs007

I think it still means the ranking function you modified has some bug. maybe you should debug into it

[JackTan25]

I feel strange yet. I think you can test the small_data I give you, when the topk2 is low(50), the recall rate is still zero, when I lift it up to 1000(dataset is 1000), it can only get 94%. Multi Vector's recall is very low, but Single Vector recall is very high and quick. cc @czs007

I think it still means the ranking function you modified has some bug. maybe you should debug into it

well, I think the recall is low,but I modify the top-k2, it can really grow the recall. The ranking function is right, maybe it's the algorithm's bug. cc @xiaofan-luan

[JackTan25]

I just modify two places here. cc @xiaofan-luan @czs007 Is this right?

[JackTan25]

@czs007 @xiaofan-luan Is there any other logic that we need to check? I'm not familiar with the code module.

[JackTan25]

well, I find a thing is that, seems for the weight rank, the score is not sum, but a single column. @czs007 Where is the logic of this part?

[JackTan25]

I feel strange yet. I think you can test the small_data I give you, when the topk2 is low(50), the recall rate is still zero, when I lift it up to 1000(dataset is 1000), it can only get 94%. Multi Vector's recall is very low, but Single Vector recall is very high and quick. cc @czs007

I think it still means the ranking function you modified has some bug. maybe you should debug into it

well, I think the recall is low,but I modify the top-k2, it can really grow the recall. The ranking function is right, maybe it's the algorithm's bug. cc @xiaofan-luan

grow to 1000, can get 100%. cc @xiaofan-luan

[JackTan25]

The code here is very strange here. What does the meaning of realTopK? The limit user gives or the number of vector column? cc @czs007 @xiaofan-luan

[yanliang567]

any updates?

[xiaofan-luan]

The code here is very strange here. What does the meaning of realTopK? The limit user gives or the number of vector column? cc @czs007 @xiaofan-luan

I guess real topk means topk. but sometimes a search can not return limit result(For example, you ask for topk1000 but there are only 500 entities in milvus, then real topk is 500),