Load collection slowly and high disk usage

[gaozh424]

I am currently using version 2.3.14 of the standalone , and I have created 10 collections with 512 dims, having inserted a total of approximately 400 million records. After completing the index creation, the disk space used on MinIO is 843 GB.

When I load all the collections, I encounter two issues:

The loading speed for the collections is very slow. It took over an hour to load 10 collections, even though I am using an SSD.

During the loading process, the disk usage continued to increase, eventually reaching about 1.7 TB, roughly double the total file size of MinIO, and it did not release until I completely shut down Milvus.

Are these issues normal? How can I speed up the collection loading process? Isn't the loading process supposed to transfer data from the disk to memory? Why does the disk usage keep increasing?

[xiaofan-luan]

I am currently using version 2.3.14 of the standalone , and I have created 10 collections with 512 dims, having inserted a total of approximately 400 million records. After completing the index creation, the disk space used on MinIO is 843 GB.

When I load all the collections, I encounter two issues:

The loading speed for the collections is very slow. It took over an hour to load 10 collections, even though I am using an SSD.

During the loading process, the disk usage continued to increase, eventually reaching about 1.7 TB, roughly double the total file size of MinIO, and it did not release until I completely shut down Milvus.

Are these issues normal? How can I speed up the collection loading process? Isn't the loading process supposed to transfer data from the disk to memory? Why does the disk usage keep increasing?

Hi @gaozh424
could you share you config and which directory takes most of your disk? and also what disk you created

For 400 million data, the data it self is almost 800GB, I believe minio should at least has double data size(Index and data)? so 1.7TB minio usage makes a lot of sense to me.

[gaozh424]

I export an env : DOCKER_VOLUME_DIRECTORY=/data , this is the du result in dir /data/volumes:

[gaozh424]

But the df shows different:

[gaozh424]

After I stop milvus， df and du become same 。 Here is the attu dispaly， Because the disk is not enough， not load all the collections。

[gaozh424]

[xiaofan-luan]

and remember minio make take extra disk for erasure coding, check https://min.io/docs/minio/linux/operations/concepts/erasure-coding.html

[gaozh424]

Any suggestions? Does the loading speed of the collection meet expectations?

[xiaofan-luan]

1. check the disk, cpu usage and bandwidth of your cluster, make sure any of the resource is not the bottle neck.
2. if no resource is the bottle neck, you have to parse the log see where is the bottle neck
3. check the config, queryCoord.taskMergeCap set to 1, queryCoord.taskExecutionCap set to 256

[gaozh424]

Hello, I finally found that it was a version issue. When I rolled back to version V2.3.4, everything worked normally. The load was very fast and completed in just a few minutes, and there was no increase in disk usage. I'm not sure why, but I will be cautious about upgrading versions in the future.

[xiaofan-luan]

could you try
"queryNode.cache.warmup" -> off
see if it works?

[gaozh424]

Thanks ,I will try it later

[gaozh424]

Hi, I tried it out and this option works. The loading speed of the collection has significantly increased, although the disk space has also increased, but the growth rate has decreased to 1/4 of the original size. Can you explain why this is?