[Bug]: The results returned by the count(*) are inaccurate and keep changing

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4-20240618-79546a6c-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

create milvus with config:

  config:
    dataCoord:
      segment:
        sealProportion: 1.52e-05
    log:
      level: debug
    quotaAndLimits:
      flushRate:
        enabled: true
        max: 0.1 
    trace:
      exporter: jaeger
      jaeger:
        url: http://tempo-distributor.tempo:14268/api/traces

test steps

1. create collection with 1024 partitions (partition-key), 1 shard
2. create index
3. insert 10m-128d data -> flush
4. index -> load
5. concurrent requests: search + upsert + flush
   
6. expected count(*): 10m, actually:

connections.disconnect("default")
connections.connect(host="10.104.13.204")
utility.list_collections()
['fouram_6xJhw13C']
c = Collection(name='fouram_6xJhw13C')
c.query('id >=0', output_fields=["count(*)"], consistency_level="Strong")
data: ["{'count(*)': 10012932}"] ..., extra_info: {'cost': 0}
c.query('id >=0', output_fields=["count(*)"], consistency_level="Strong")
data: ["{'count(*)': 10012407}"] ..., extra_info: {'cost': 0}
c.name
'fouram_6xJhw13C'
c.query('id >=0', output_fields=["count(*)"], consistency_level="Strong")
data: ["{'count(*)': 10012372}"] ..., extra_info: {'cost': 0}
c.query('id >=0', output_fields=["count(*)"], consistency_level="Strong")
data: ["{'count(*)': 10012294}"] ..., extra_info: {'cost': 0}
c.query('id >=0', output_fields=["count(*)"], consistency_level="Strong")
data: ["{'count(*)': 10012261}"] ..., extra_info: {'cost': 0}

Expected Behavior

No response

Steps To Reproduce

https://argo-workflows.zilliz.cc/archived-workflows/qa/88b56c6a-eb3d-4862-95a6-b0c64434efde?nodeId=compact-opt-1024-with-flush-2

Milvus Log

• grafana: compact-opt-flush2
  pods:

compact-opt-flush2-milvus-datanode-5898b9d778-sshqx               1/1     Running     0                82m     10.104.5.70     4am-node12   <none>           <none>
compact-opt-flush2-milvus-indexnode-8c577d9d6-9tnms               1/1     Running     0                82m     10.104.17.163   4am-node23   <none>           <none>
compact-opt-flush2-milvus-indexnode-8c577d9d6-9wl8n               1/1     Running     0                82m     10.104.6.58     4am-node13   <none>           <none>
compact-opt-flush2-milvus-indexnode-8c577d9d6-qq9c4               1/1     Running     0                82m     10.104.20.226   4am-node22   <none>           <none>
compact-opt-flush2-milvus-mixcoord-5b9f79b984-zwfn2               1/1     Running     0                82m     10.104.4.88     4am-node11   <none>           <none>
compact-opt-flush2-milvus-proxy-b55c6db47-vnzc2                   1/1     Running     0                82m     10.104.13.204   4am-node16   <none>           <none>
compact-opt-flush2-milvus-querynode-0-786c99d5cc-k4bcz            1/1     Running     0                82m     10.104.18.196   4am-node25   <none>           <none>
compact-opt-flush2-milvus-querynode-0-786c99d5cc-q5znj            1/1     Running     0                82m     10.104.13.205   4am-node16   <none>           <none>

Anything else?

No response

[yanliang567]

/unassign

[sunby]

I've identified a potential issue. If a segment's DML position is larger than the earlist growing segment's start position, it might not be included in the L0 compaction, resulting in some data not being deleted.

[XuanYang-cn]

One more possible case:
#33907 Fixed a bug about deletion been written into wrong partitions, causing some insertdata not getting deleted correctly, so count(*) increased.

The commit id was: e807183

[XuanYang-cn]

I've identified a potential issue. If a segment's DML position is larger than the earlist growing segment's start position, it might not be included in the L0 compaction, resulting in some data not being deleted.

Good Catch, but very rare for there need to be 2 growing segments existing at the same time and the latter one is sealed before the eariler one.

[ThreadDao]

@sunby cherry-pick fix pr to 2.4 branch?