[Bug]: Missing data in hybrid search result when using partition key

[IsaacWhittakerTR]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.3.7 (upgraded recently from 2.3.2)
- Deployment mode(standalone or cluster): EKS Cluster
- MQ type(rocksmq, pulsar or kafka): Kafka
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.3.6
- OS(Ubuntu or CentOS): n/a - EKS
- CPU/Memory: 
- GPU: n/a
- Others: s3 for data persistance

Current Behavior

When performing a hybrid search for vectors which exist in collection and filtering on the partition key, either an empty result is returned or a result with a few distant vectors are returned IF a boolean expression with the partition_key in [123, 456] TermExpr syntax is used.

The correct results are returned if the boolean expression uses the partition_key == 123 or partition_key == 456 CmpOp is used instead. Correct results are also returned when filtering on any of the non-partition key scalar fields.****

Expected Behavior

When performing a hybrid search on a vector that exists in the collection, and filtering by the partition key which is associated with that vector, the search vector should be returned as one of the results with a zero distance.

Steps To Reproduce

1. Query a vector to confirm it exists in the collection, with `partition_key` 123.
2. Perform a hybrid search on that vector with `partition_key in [123]` filter.
3. Query returns an empty result.

Milvus Log

I was not able to find any related error or warning logs, but I can provide the logs on request.

I could not find any issues with the collection using birdwatcher either.

Anything else?

Schema information:

field_1 -> type int32 with scalar index
field_2 -> partition key, type int64 with scalar index
field_3 -> primary key, type int64
field_4 -> type int32 with scalar index
field_5 -> type FloatVector with dim 768 and IVF_SQ8 index (metric type L2, nlist 8192)

A Zilliz user mentioned a very similar problem in this discord thread: https://discord.com/channels/1160323594396635310/1201965988451454996

[IsaacWhittakerTR]

Update - First downgrading to Milvus v2.3.2 and pymilvus v2.3.2, then re-upserting specific entities seems to make those vectors suddenly searchable again with any of the hybrid search filtering syntaxes mentioned above.

The issue appears to be specific to Milvus or pymilvus greater than version 2.3.2 - unfortunately this means we will need to downgrade our cluster and re-upsert 100 Million+ vectors for search to work again which is obviously less than ideal. Any idea what could have changed in the new versions to break hybrid search?

[xiaofan-luan]

Seems to be a potential bug introduced recently.

@yanliang567
please help on it.

[xiaofan-luan]

if this is a bug, pls assign to @zhagnlu

[yanliang567]

trying to reproduce it in house

[yanliang567]

@IsaacWhittakerTR I failed to reproduce the issue in house. could you please reproduce the issue and collect the milvus logs for investigation?

I roughly tried below on milvus v2.3.8

1. create a collection
   1.1 with partition key, type int64 with scalar index(named category)
   1.2 with type FloatVector(named embedding) with dim 512 and IVF_SQ8 index (metric type L2, nlist 1024)
2. insert 2 million entities
3. run a query to confirm the vec exits in the collection, category==1
4. search
   res1=c.search(data=[vec], anns_field="embedding", param={}, limit=10, expr="category in [1]")
   res2=c.search(data=[vec], anns_field="embedding", param={}, limit=10, expr="category==1")
   res1 equals res2

[yanliang567]

/assign @IsaacWhittakerTR
please reproduce a search with empty result, then refer this doc to export the whole Milvus logs. Thx

[zhagnlu]

I also failed to reproduce this error using 2.3.7. please provide more detailed infos.

[IsaacWhittakerTR]

Thank you both for trying to reproduce the issue. Did either of you try creating a collection first using an older version of Milvus (i.e. v2.3.1 or v2.3.2), and then upgrading the cluster to a newer version (2.3.6) and inserting vectors after upgrading?

This is the exact situation which caused issues for us. Downgrading the cluster to v2.3.2 and re-upserting vectors fixed the issue, and then upgrading back to v2.3.7 and re-upserting the vectors would consistantly give empty results for hybrid searches on our partition key.

I will try to reproduce the issue again and export the full Milvus logs. Is there by chance an e-mail address I can send the logs to instead of uploading them directly here?

[xiaofan-luan]

sounds like this could be a compatibility issue?

[yanliang567]

@IsaacWhittakerTR thank you for your patience, and I will try to reproduce it with an upgrade. If you have logs, please mail to me: yanliang.qiao@zilliz.com

[yanliang567]

@IsaacWhittakerTR I tried today with an upgrade from v2.3.1 to v2.3.7, but no luck. So please share the logs when you got them. thx.

[IsaacWhittakerTR]

@yanliang567 Thanks again for your continued assistance on this issue. It is strange to me that it is not reproducible for you as it is very consistent on my end (but I can't think what else would differ between our setup that would change the behavior of the partitioning logic).

I sent over our logs via e-mail, here is a detailed list of the steps I took to reproduce the incorrect search results:

1. Upgrade our cluster from Milvus v2.3.2 to v2.3.7 via helm upgrade.
2. Query the cluster to get vector and metadata for an existing entity (with primary key value 1286322).
3. Upsert that entity to the existing collection using pymilvus v2.3.6.
4. Query again using <primary_key> in [1286322] to confirm the vector still exists in the collection.
5. Perform several searches using this vector:

• Hybrid search with TermExpr on partition key does not return the entity in results
• Hybrid search with CmpOp on partition key does not return the entity in results
• Hybrid search using another non-partition key correctly returns the entity with primary key 1286322 with distance 0.0
• Regular search with no filter correctly returns the entity with primary key 1286322 in results with distance 0.0

[Richard-lrg]

I also reproduced the same problem
The issue occurred when the milvus cluster was upgraded from 2.3.3 to 2.3.9.
The steps to reproduce the problem are as follows:
Query expr： id in ["l6UDMYe17zSECY"] return the expected results

When adding conditions, partitionKey in ["xxx"] and id in ["l6UDMYe17zSECY"] have no results (it can be confirmed that the relevant field values are correct)

[yanliang567]

@Cactus-L @IsaacWhittakerTR thank you both for updates. We will look into the logs and keep you posted.

[yanliang567]

@Cactus-L could you please share your collection schema and index params for me to trying a reproduction?

[Richard-lrg]

@Cactus-L could you please share your collection schema and index params for me to trying a reproduction?

is this info still needed?
I see a fix code has been submitted.

[yanliang567]

yep...we are testing on the fix pr, now will release a new version soon after verification done. Thank you both for popping this issue up. @Cactus-L @IsaacWhittakerTR

[czs007]

fix pr :#30773
please wait for the new version 2.3.10

[emaestre]

yep...we are testing on the fix pr, now will release a new version soon after verification done. Thank you both for popping this issue up. @Cactus-L @IsaacWhittakerTR

Glad to hear that the issue is solved! Great job guys 🙌🏻

Will be attentive for the release of the new version to update it on our end as well.

[xiaofan-luan]

we have already released 2.3.10.
The problem is solved on the new release.
We are also working on a fix about the existing problem data

[yanliang567]

I'd close this issue as I have verified the fix on milvus 2.3.10