[Bug]: [CI] Deleted data can be queried

[NicoYuan1986]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-20240516-5b27a0cd
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Deleted data can be queried.

[2024-05-17T14:14:54.616Z] [2024-05-17 13:36:11 - DEBUG - ci_test]: (api_request)  : [Collection.delete] args: ['int64 in [0]', None, 180], kwargs: {} (api_request.py:62)
[2024-05-17T14:14:54.616Z] [2024-05-17 13:36:11 - DEBUG - ci_test]: (api_response) : (insert count: 0, delete count: 1, upsert count: 0, timestamp: 0, success count: 0, err count: 0, cost: 0)  (api_request.py:37)
[2024-05-17T14:14:54.616Z] [2024-05-17 13:36:11 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)
[2024-05-17T14:14:54.616Z] [2024-05-17 13:36:17 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-05-17T14:14:54.616Z] [2024-05-17 13:36:17 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'FLAT', 'params': {}, 'metric_type': 'COSINE'}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-05-17T14:14:54.616Z] [2024-05-17 13:36:17 - DEBUG - ci_test]: (api_response) : Status(code=0, message=)  (api_request.py:37)
[2024-05-17T14:14:54.616Z] [2024-05-17 13:36:17 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-05-17T14:14:54.616Z] [2024-05-17 13:36:22 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-05-17T14:14:54.616Z] [2024-05-17 13:36:22 - DEBUG - ci_test]: (api_request)  : [Collection.query] args: ['int64 in [0]', None, None, 180], kwargs: {} (api_request.py:62)
[2024-05-17T14:14:54.616Z] [2024-05-17 13:36:23 - DEBUG - ci_test]: (api_response) : data: ["{'int64': 0}"] ..., extra_info: {'cost': 0}  (api_request.py:37)

Expected Behavior

query result is empty.

Steps To Reproduce

No response

Milvus Log

1. link: https://qa-jenkins.milvus.io/blue/organizations/jenkins/E2E%20Test/detail/E2E%20Test/474/pipeline/
2. log: artifacts-e2e-test-474-server-logs.tar.gz
3. collection name: delete_HrUnwOKN

Anything else?

No response

[NicoYuan1986]

reproduced in nightly (one pod). master-20240521-3d105fcb-arm64

1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/master/749/pipeline/159/
2. log: artifacts-milvus-standalone-one-pod-nightly-749-pymilvus-e2e-logs.tar.gz
3. failed time: [2024-05-21T18:07:07.069Z] [gw5] [ 20%] FAILED testcases/test_compaction.py::TestCompactionParams::test_compact_after_delete
4. collection name: compact_WSUd1Vt9

log:

[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:18 - DEBUG - ci_test]: (api_request)  : [Collection.delete] args: ['int64 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74......, kwargs: {} (api_request.py:62)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:18 - DEBUG - ci_test]: (api_response) : (insert count: 0, delete count: 50000, upsert count: 0, timestamp: 0, success count: 0, err count: 0, cost: 0)  (api_request.py:37)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:18 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:19 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:19 - DEBUG - ci_test]: (api_request)  : [Collection.compact] args: [180], kwargs: {} (api_request.py:62)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:19 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:24 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:30 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:30 - DEBUG - ci_test]: (api_request)  : [Collection.query] args: ['int64 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74......, kwargs: {} (api_request.py:62)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:30 - DEBUG - ci_test]: (api_response) : data: [] , extra_info: {'cost': 0}  (api_request.py:37)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:30 - DEBUG - ci_test]: (api_request)  : [Collection.release] args: [], kwargs: {'timeout': 180} (api_request.py:62)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:06:31 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:07:01 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:07:06 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:07:06 - DEBUG - ci_test]: (api_request)  : [Collection.query] args: ['int64 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74......, kwargs: {} (api_request.py:62)
[2024-05-21T19:54:05.259Z] [2024-05-21 18:07:07 - DEBUG - ci_test]: (api_response) : data: ["{'int64': 0}", "{'int64': 1}", "{'int64': 2}", "{'int64': 3}", "{'int64': 4}", "{'int64': 5}", "{'int64': 6}", "{'int64': 7}", "{'int64': 8}", "{'int64': 9}"] ..., extra_info: {'cost': 0}  (api_request.py:37)

steps:

1. delete data, flush , compact
2. load, query -> 0 result
3. release, load, query -> ❌ deleted data

[NicoYuan1986]

keep reproducing.

[aoiasd]

Sync manager remove sync task before flush task finish, cause push checkpoint forward earlier.
So L0 segment not flush finished but checkpoint pushed, cause load loss delete data.
Will fixed by @congqixia

[NicoYuan1986]

fixed.