Memory consumed totally 3-4 times as data size

[yanliang567]

Steps/Code to reproduce:

in tests20/python_client: run pytest test_e2e.py --host x.x.x.x after updating nb=100,000
or

1. deploy Milvus 2.0 cluster
   1.1 after deployment, Milvus consumes about 380MB memory totally.
2. insert 100,000 entities (about 50MB) into a collection
3. flush
4. load
5. search
6. make a sq8 index
7. query

Expected result:

memory consumed smoothly

Actual results:

total memory consumed from 380MB to 1480MB and keeps at 750MB after step#7, looking into different pods:
querynode: 50MB to 222MB, and keeps at 200MB after step#7
datanode: 50MB to 470MB, and keeps 152MB after step#7
Indexnode: 50M to 286MB, and keeps at 150MB after step#7
proxy: 60M to 336MB, and keeps at 75MB after step#7 (I think 75MB is acceptable after step#7)

Environment:

• Milvus version(e.g. v2.0.0-RC2 or 8b23a93): master-20210810-bdb8396
• Deployment mode(standalone or cluster): cluster
• SDK version(e.g. pymilvus v2.0.0rc2):
  pymilvus 2.0.0rc3.dev16
  pymilvus-orm 2.0.0rc3.dev15

Configuration file:

Additional context:

after deployment with no workload:

when there is workload

after step#7

[yanliang567]

there is a similar issue which focuses on querynode, #6745

[congqixia]

Currently invoking cgo functionality must do data copy, which means same data occupies twice memory in both golang and cpp side.
And decoding message and row-column conversion consumes same amount of memory as well.
That's the two main reasons cause 3~4 times memory consumption.
Maybe we should have some solution for this case.
@xige-16

[xige-16]

Currently invoking cgo functionality must do data copy, which means same data occupies twice memory in both golang and cpp side.
And decoding message and row-column conversion consumes same amount of memory as well.
That's the two main reasons cause 3~4 times memory consumption.
Maybe we should have some solution for this case.
@xige-16

Changing the format of the internal data to arrow will solves the problem of copying the data between cgo and golang.

[congqixia]

An MEP is in WIP status for this. Please check #7210

[xiaofan-luan]

assign @cydrain

[xiaofan-luan]

why are we copying the data? from design perspective, data should not be resident in go?

[congqixia]

why are we copying the data? from design perspective, data should not be resident in go?

By copying the data, do you mean copying the data from go managed memory to c managed memory? If that's the case, this behavior is the designed behavior of cgo. The reason is that it's unsafe to allow c/cpp to operate memory from go runtime. Say if the memory is cleaned up by gc when c/cpp tried to operate it.

[xige-16]

I will check if there is a memory leak

[cydrain]

/assign

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[yanliang567]

lets keep it