[Bug]: The memory for loading tens of millions of sets has been stuck for over 2 hours

[stqf]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.3.18 
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): OpenEuler 22.03 LTS-SP2
- CPU/Memory: 64C aarch64 / 512GiB
- GPU: N/A
- Others: 58TiB HDD

Current Behavior

The Milvus service got stuck when it was restarted

Collection structure follows

Expected Behavior

1. It can be loaded successfully and quickly

2. If the loading fails, there is a clear log for easy positioning

Steps To Reproduce

1. Simulate the basic data of 2kw+

2. Restart the Milvus service

3. Repeatedly load and release the collection

Milvus Log

loading.20250421152000.log

Anything else?

I would be extremely grateful if you could offer me much advice

[xiaofan-luan]

[2025/04/21 07:15:51.912 +00:00] [WARN] [task/executor.go:246] ["failed to load segment"] [taskID=1745219139849] [collectionID=456003875677388362] [replicaID=457498554850869249] [segmentID=456547787876408447] [node=18] [source=segment_checker] [shardLeader=18] [error="failed to read files/stats_log/456003875677388362/456547787875064198/456547787876408447/100/1: attempt #0: IO failed[key=files/stats_log/456003875677388362/456547787875064198/456547787876408447/100/1]: Resource requested is unreadable, please reduce your request rate"]

it seems that your storage system is slow.

please make sure you are not using nfs for your underlying storage

[xiaofan-luan]

and 2.3.18 is pretty old .

We recommend to use 2.4.23 or 2.5.10

[stqf]

[2025/04/21 07:15:51.912 +00:00] [WARN] [task/executor.go:246] ["failed to load segment"] [taskID=1745219139849] [collectionID=456003875677388362] [replicaID=457498554850869249] [segmentID=456547787876408447] [node=18] [source=segment_checker] [shardLeader=18] [error="failed to read files/stats_log/456003875677388362/456547787875064198/456547787876408447/100/1: attempt #0: IO failed[key=files/stats_log/456003875677388362/456547787875064198/456547787876408447/100/1]: Resource requested is unreadable, please reduce your request rate"]

it seems that your storage system is slow.

please make sure you are not using nfs for your underlying storage

[xiaofan-luan]

@stqf
could you check the iostats or sar?
Resource requested is unreadable meaning too many iops is sent to storage.

HDD is too slow as well, maybe you can reduce the size of the thread so load is slower but can be done.

Also upgrade to latest milvus will help as well.

[xiaofan-luan]

with hdd your disk is slow (with hundred iops) but your machine is huge so milvus enabled 500+ thread to concurrently load.

in your case upgrade to 2.4.23 and reduce the thread number to 2 * core number and see if it can work

[stqf]

could you check the iostats or sar? Resource requested is unreadable meaning too many iops is sent to storage.

HDD is too slow as well, maybe you can reduce the size of the thread so load is slower but can be done.

Also upgrade to latest milvus will help as well.

Yes, the read IOPS is relatively high, about 500+. I tried to limit the CPU through docker and verified the CPU usage limit successfully through docker stats, but the loading is still stuck. Is there any configuration item that can explicitly declare the corresponding number of threads? Because my application scenario is that the concurrency of writing and querying is not large, but the incremental data volume over time is very large. After an unexpected restart, the loading fails (gets stuck), and the service is unavailable.

[stqf]

with hdd your disk is slow (with hundred iops) but your machine is huge so milvus enabled 500+ thread to concurrently load.

in your case upgrade to 2.4.23 and reduce the thread number to 2 * core number and see if it can work

1. The production environment is not easy to upgrade, and the verification cycle is a bit long
2. How to control the number of concurrent threads during loading, preferably through environment variables
3. The ERROR level in the log says that there is data corruption, how to fix it

[xiaofan-luan]

2. ber of concurrent threads during loading, preferably through environment variables
3. The ERROR level in the log says that there is data corruption, how to fix it

We also noticed data corruption.
Have you checked for potential disk issues? It’s possible the device itself is faulty. For production environments, we strongly recommend using S3 or at least EBS for storage.

If there's only a single replica and the device fails, data loss is inevitable—and unfortunately, there's nothing we can do to recover it in that case.

[yanliang567]

/assign @stqf
/unassign