[Bug]: [string_match] The query results for all the string match are empty when the string field is primary key with INVERTED index created after flush

[binbinlv]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-20240218-99297ab8
- Deployment mode(standalone or cluster): both
- MQ type(rocksmq, pulsar or kafka):    all
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc35
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The query results for all the string match are empty when the string field is primary key with INVERTED index created after flush

>>> res = collection.query("string like '%0'")
>>> len(res)
200
>>> res = collection.query("string like '%0%'")
>>> len(res)
452
>>> collection.flush()
>>>
>>>
>>> res = collection.query("string like '0%'")
>>> len(res)
1
>>> res
[{'string': '0'}]
>>>
>>>
>>> collection.flush()
>>>
>>> res = collection.query("string like '0%'")
>>> len(res)
0
>>>
>>>
>>> res = collection.query("string like '%0'")
>>> len(res)
0
>>> res = collection.query("string like '%0%'")
>>> len(res)
0
>>> res
[]

Expected Behavior

query successfully with correct ids

Steps To Reproduce

from pymilvus import CollectionSchema, FieldSchema
from pymilvus import Collection
from pymilvus import connections
from pymilvus import DataType
from pymilvus import Partition
from pymilvus import utility
connections.connect()

dim = 128
string_field = FieldSchema(name="string", dtype=DataType.VARCHAR, max_length=65535, is_primary=True)
int64_field = FieldSchema(name="int64", dtype=DataType.INT64)
float_field = FieldSchema(name="float", dtype=DataType.FLOAT)
bool_field = FieldSchema(name="bool", dtype=DataType.BOOL)
float_vector = FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
schema = CollectionSchema(fields=[string_field, float_field, bool_field, int64_field, float_vector])

import numpy as np
collection = Collection("test_search_collection_binbin_tmp_0", schema=schema)
nb = 2000
import random
default_search_params = {"metric_type": "L2", "params": {"nprobe": 100}}
vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
res = collection.insert([[str(i) for i in range(nb)], [np.float32(i) for i in range(nb)], [np.bool_(i) for i in range(nb)], [i for i in range(nb)], vectors])
index_param = {"index_type": "IVF_FLAT", "metric_type": "L2", "params": {"nlist": 100}}
collection.create_index("float_vector", index_param, index_name="a")
index_param = {"index_type": "INVERTED"}
collection.create_index("string", index_param, index_name="b")
collection.load()
res = collection.query("string like '0%'")
len(res)
res = collection.query("string like '%0'")
len(res)
res = collection.query("string like '%0%'")
len(res)
collection.flush()
res = collection.query("string like '0%'")
len(res)
collection.flush()
res = collection.query("string like '0%'")
len(res)
res = collection.query("string like '%0'")
len(res)
res = collection.query("string like '%0%'")
len(res)

Milvus Log

No response

Anything else?

No response

[longjiquan]

same with #30687, and already fixed by #30764
/assign @binbinlv

[binbinlv]

It is not fixed, maybe not the same root cause.

milvus:master-20240222-16b4c9a7
pymilvus: 2.4.0rc35

results:

>>> dim = 128
>>> string_field = FieldSchema(name="string", dtype=DataType.VARCHAR, max_length=65535, is_primary=True)
>>> int64_field = FieldSchema(name="int64", dtype=DataType.INT64)
>>> float_field = FieldSchema(name="float", dtype=DataType.FLOAT)
>>> bool_field = FieldSchema(name="bool", dtype=DataType.BOOL)
>>> float_vector = FieldSchema(name="float_vector", dtype=DataType.FLOAT_VECTOR, dim=dim)
>>> schema = CollectionSchema(fields=[string_field, float_field, bool_field, int64_field, float_vector])
>>> import numpy as np
>>> collection = Collection("test_search_collection_30728", schema=schema)
>>> nb = 2000
>>> import random
>>> default_search_params = {"metric_type": "L2", "params": {"nprobe": 100}}
>>> vectors = [[random.random() for _ in range(dim)] for _ in range(nb)]
>>> res = collection.insert([[str(i) for i in range(nb)], [np.float32(i) for i in range(nb)], [np.bool_(i) for i in range(nb)], [i for i in range(nb)], vectors])
>>> index_param = {"index_type": "IVF_FLAT", "metric_type": "L2", "params": {"nlist": 100}}
>>> collection.create_index("float_vector", index_param, index_name="a")
Status(code=0, message=)
>>> index_param = {"index_type": "INVERTED"}
>>> collection.create_index("string", index_param, index_name="b")
Status(code=0, message=)
>>> collection.load()
>>> res = collection.query("string like '0%'")
>>> len(res)
1
>>> res = collection.query("string like '%0'")
>>> len(res)
200
>>> res = collection.query("string like '%0%'")
>>> len(res)
452
>>> collection.flush()
>>> res = collection.query("string like '0%'")
>>> len(res)
1
>>> collection.flush()
>>> res = collection.query("string like '0%'")
>>> len(res)
1
>>> collection.flush()
>>> collection.flush()
>>> collection.flush()
>>> res = collection.query("string like '%0'")
>>> len(res)
0
>>> res = collection.query("string like '%0%'")
>>> len(res)
0

[binbinlv]

/unassign

[longjiquan]

should be fixed by #30848

[longjiquan]

/assign @binbinlv

[binbinlv]

Verified and fixed:

milvus: master-20240228-095cdbed-amd64
pymilvus: 2.4.0rc37
results:

>>> len(res)
1
>>> res = collection.query("string like '%0'")
>>> len(res)
200
>>> res = collection.query("string like '%0%'")
>>> len(res)
452
>>> collection.flush()
>>> res = collection.query("string like '0%'")
>>> len(res)
1
>>> collection.flush()
>>> res = collection.query("string like '0%'")
>>> len(res)
1
>>> collection.flush()
>>> collection.flush()
>>> collection.flush()
>>> res = collection.query("string like '%0'")
>>> len(res)
200
>>> res = collection.query("string like '0%'")
>>> len(res)
1
>>>
>>> res = collection.query("string like '%0%'")
>>> len(res)
452
>>> collection.flush()
>>> collection.flush()
>>> collection.flush()
>>> collection.flush()
>>> collection.flush()
>>> collection.flush()
>>> collection.flush()
>>> collection.flush()
>>> collection.flush()
>>>
>>>
>>>
>>>
>>>
>>> res = collection.query("string like '0%'")
>>> len(res)
1
>>> res = collection.query("string like '%0'")
>>> len(res)
200
>>> res = collection.query("string like '%0%'")
>>> len(res)
452