[CI_Block]num_entities of partition is not the same with inserted quantities under multiple replicas

[binbinlv]

Please state your issue using the following template and, most importantly, in English.

Describe the bug
num_entities of partition is not the same with inserted quantity under multiple replicas when data is relatively big

Steps/Code to reproduce behavior
1 deploy multiple replicas enviroment:
proxy: 1, querynode:2, indexnode:2, datanode: 2

2 run test case: tests20/python_client/testcases/test_partition.py::TestPartitionOperations::test_partition_insert_maximum_size_data

Result:
assert partition_w.num_entities == max_size
E assert 77090 == 100000
E +77090
E -100000

Logs:
log.tar.zip

Expected behavior
num_entities of partition equals to inserted quantities

Method of installation

• Docker/cpu
• Docker/gpu
• Build from source

Environment details

• Hardware/Software conditions (OS, CPU, GPU, Memory)

• Milvus version (master or released version)
  milvus-master (a8e5fd2)
  Name: pymilvus
  Version: 2.0.0rc2.dev11
  Name: pymilvus-orm
  Version: 2.0.0rc2.dev29

Configuration file
Settings you made in server_config.yaml or milvus.yaml

paste-file-content-here

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[yanliang567]

in CI tests(no replicas), if insert 100,000 entities, it repros as well:

# insert data to partition
      max_size = 100000  # TODO: clarify the max size of data
      ins_res, succ = partition_w.insert(cf.gen_default_dataframe_data(max_size))
      assert len(ins_res.primary_keys) == max_size
      assert succ
      # self._connect().flush([collection_w.name])
>       assert partition_w.num_entities == max_size
E       assert 98618 == 100000

It looks loke flush is not completed when calling .num_entities. Does flush run as sync or async by default?

[yanliang567]

it seems to be getting worse and worse, this time it repros by 3000 entities in CI tests(only in standalone pipeline):

stats = connect.get_collection_stats(collection_list[i])=
[2021-07-16T08:14:07.991Z] > assert stats[row_count] == default_nb
[2021-07-16T08:14:07.991Z] E assert 2064 == 3000
[2021-07-16T08:14:07.991Z] E +2064
[2021-07-16T08:14:07.991Z] E -3000
[2021-07-16T08:14:07.991Z]
[2021-07-16T08:14:07.991Z] collection/test_collection_stats.py:370: AssertionError
[2021-07-16T08:14:07.991Z] =========================== short test summary info ============================
[2021-07-16T08:14:07.991Z] FAILED collection/test_collection_stats.py::TestGetCollectionStats::test_collection_count_multi_collections

[yanliang567]

@czs007 any updates, dudes?

[del-zhenwu]

Also re-pro with multi collections

[2021-07-23T02:39:33.993Z] self = <test_collection_count.TestCollectionMultiCollections object at 0x7faf593a10b8>

[2021-07-23T02:39:33.993Z] connect = <pymilvus.client.stub.Milvus object at 0x7faf5abca630>

[2021-07-23T02:39:33.993Z] 

[2021-07-23T02:39:33.993Z]     @pytest.mark.level(2)

[2021-07-23T02:39:33.993Z]     @pytest.mark.tags(CaseLabel.tags_smoke)

[2021-07-23T02:39:33.993Z]     def test_collection_count_multi_collections_mix(self, connect):

[2021-07-23T02:39:33.993Z]         '''

[2021-07-23T02:39:33.993Z]         target: test collection rows_count is correct or not with multiple collections of JACCARD

[2021-07-23T02:39:33.993Z]         method: create collection and add entities in it,

[2021-07-23T02:39:33.993Z]             assert the value returned by count_entities method is equal to length of entities

[2021-07-23T02:39:33.993Z]         expected: the count is equal to the length of entities

[2021-07-23T02:39:33.993Z]         '''

[2021-07-23T02:39:33.993Z]         collection_list = []

[2021-07-23T02:39:33.993Z]         collection_num = 20

[2021-07-23T02:39:33.993Z]         for i in range(0, int(collection_num / 2)):

[2021-07-23T02:39:33.993Z]             collection_name = gen_unique_str(uid)

[2021-07-23T02:39:33.993Z]             collection_list.append(collection_name)

[2021-07-23T02:39:33.993Z]             connect.create_collection(collection_name, default_fields)

[2021-07-23T02:39:33.993Z]             connect.insert(collection_name, default_entities)

[2021-07-23T02:39:33.993Z]         for i in range(int(collection_num / 2), collection_num):

[2021-07-23T02:39:33.993Z]             collection_name = gen_unique_str(uid)

[2021-07-23T02:39:33.993Z]             collection_list.append(collection_name)

[2021-07-23T02:39:33.993Z]             connect.create_collection(collection_name, default_binary_fields)

[2021-07-23T02:39:33.993Z]             res = connect.insert(collection_name, default_binary_entities)

[2021-07-23T02:39:33.993Z]         connect.flush(collection_list)

[2021-07-23T02:39:33.993Z]         for i in range(collection_num):

[2021-07-23T02:39:33.993Z]             stats = connect.get_collection_stats(collection_list[i])

[2021-07-23T02:39:33.993Z] >           assert stats[row_count] == default_nb

[2021-07-23T02:39:33.993Z] E           assert 2064 == 3000

[2021-07-23T02:39:33.993Z] E             +2064

[2021-07-23T02:39:33.993Z] E             -3000

[2021-07-23T02:39:33.993Z] 

[2021-07-23T02:39:33.993Z] collection/test_collection_count.py:499: AssertionError

[2021-07-23T02:39:33.993Z] =========================== short test summary info ============================

[2021-07-23T02:39:33.993Z] FAILED collection/test_collection_count.py::TestCollectionMultiCollections::test_collection_count_multi_collections_mix

[czs007]

I am working on it.