100个Segment段执行合并的操作的时候，集群不稳定.minio写入频繁出现InvalidPart - multipart upload

[eyeeye]

4台物理机 32C 96G。k8s+helm部署了所有依赖的节点
100个segment目标成10个segment .单个在10G左右

[13:54:32root@node1 /root/minio-debug-20260904-1015]
@#kubectl -n milvus logs $D | grep -v "query slots done" | tail -80
strict-transport-security : max-age=31536000; includeSubDomains
vary : Origin
x-amz-id-2 : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-request-id : 18D207FF85FA5CFE
x-content-type-options : nosniff
x-xss-protection : 1; mode=block
[WARN] 2026-09-04 05:46:56.087 AWSClient [139744665658944] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[WARN] 2026-09-04 05:46:56.087 AWSClient [139744665658944] Request failed, now waiting 200 ms before attempting again.
[ERROR] 2026-09-04 05:47:32.100 CurlHttpClient [139744665658944] Curl returned error code 28 - Timeout was reached
[ERROR] 2026-09-04 05:47:32.100 AWSXmlClient [139744665658944] HTTP response code: -1
Resolved remote host IP address: 10.43.130.122
Request ID:
Exception name:
Error message: curlCode: 28, Timeout was reached
0 response headers:
[WARN] 2026-09-04 05:47:32.100 AWSClient [139744665658944] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[WARN] 2026-09-04 05:47:32.100 AWSClient [139744665658944] Request failed, now waiting 200 ms before attempting again.
[ERROR] 2026-09-04 05:48:06.520 CurlHttpClient [139744747451968] Curl returned error code 28 - Timeout was reached
[ERROR] 2026-09-04 05:48:06.520 AWSXmlClient [139744747451968] HTTP response code: -1
Resolved remote host IP address: 10.43.130.122
Request ID:
Exception name:
Error message: curlCode: 28, Timeout was reached
15 response headers:
accept-ranges : bytes
content-length : 14325534
content-range : bytes 4199555406-4213880939/4748972516
content-security-policy : block-all-mixed-content
content-type : application/octet-stream
date : Fri, 04 Sep 2026 05:47:50 GMT
etag : "f1c45330084db893781371282d564fd2-453"
last-modified : Thu, 03 Sep 2026 09:47:21 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Origin
x-amz-id-2 : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-request-id : 18D20810DAEC9E27
x-content-type-options : nosniff
x-xss-protection : 1; mode=block
[WARN] 2026-09-04 05:48:06.520 AWSClient [139744747451968] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[WARN] 2026-09-04 05:48:06.520 AWSClient [139744747451968] Request failed, now waiting 200 ms before attempting again.
[ERROR] 2026-09-04 05:48:26.731 CurlHttpClient [139745080899136] Curl returned error code 28 - Timeout was reached
[ERROR] 2026-09-04 05:48:26.731 AWSXmlClient [139745080899136] HTTP response code: -1
Resolved remote host IP address: 10.43.130.122
Request ID:
Exception name:
Error message: curlCode: 28, Timeout was reached
15 response headers:
accept-ranges : bytes
content-length : 14330434
content-range : bytes 4515223792-4529554225/4748972516
content-security-policy : block-all-mixed-content
content-type : application/octet-stream
date : Fri, 04 Sep 2026 05:48:10 GMT
etag : "f1c45330084db893781371282d564fd2-453"
last-modified : Thu, 03 Sep 2026 09:47:21 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Origin
x-amz-id-2 : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-request-id : 18D208158F8F2752
x-content-type-options : nosniff
x-xss-protection : 1; mode=block
[WARN] 2026-09-04 05:48:26.731 AWSClient [139745080899136] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[WARN] 2026-09-04 05:48:26.731 AWSClient [139745080899136] Request failed, now waiting 200 ms before attempting again.
[ERROR] 2026-09-04 05:48:36.943 CurlHttpClient [139745080899136] Curl returned error code 28 - Timeout was reached
[ERROR] 2026-09-04 05:48:36.943 AWSXmlClient [139745080899136] HTTP response code: -1
Resolved remote host IP address: 10.43.130.122
Request ID:
Exception name:
Error message: curlCode: 28, Timeout was reached
0 response headers:
[WARN] 2026-09-04 05:48:36.943 AWSClient [139745080899136] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
[WARN] 2026-09-04 05:48:36.943 AWSClient [139745080899136] Request failed, now waiting 200 ms before attempting again.
[2026/09/04 05:48:43.098 +00:00] [INFO] [CGO] [index/VectorMemIndex.cpp:490] ["[SERVER][Build][milvus][]CacheRawDataToMemory success, build_id: 468825455167645836"]
[2026/09/04 05:48:47.552 +00:00] [INFO] [CGO] [index/VectorMemIndex.cpp:471] ["[SERVER][BuildWithDataset][milvus][]start build memory index with KNOWHERE, build_id: 468825455167645836"]
[2026/09/04 05:48:47.553 +00:00] [INFO] [CGO] [hnsw/faiss_hnsw.cc:2099] ["[KNOWHERE][operator()][knowhere_build0] Training HNSW Index"]
[2026/09/04 05:48:47.553 +00:00] [INFO] [CGO] [hnsw/faiss_hnsw.cc:1775] ["[KNOWHERE][AddInternal][knowhere_build0] Adding 600000 rows to HNSW Index"]
[2026/09/04 05:51:17.355 +00:00] [INFO] [importv2/util.go:591] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
[2026/09/04 05:51:17.355 +00:00] [INFO] [importv2/util.go:591] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=0] [failed=0]
[13:54:34root@node1 /root/minio-debug-20260904-1015]

[yhmo]

这几行信息暗示对 4GB级别大对象的读操作频繁，MinIO 已扛不住请求超时(milvus.yaml里minio.requestTimeoutMs默认10秒)，并且不断地在重试。

Curl returned error code 28 - Timeout was reached
Content-range : bytes 4199555406-4213880939/4748972516
Request failed, now waiting 200 ms before attempting again.

然后同时在读向量数据构建索引，和compaction任务抢minio I/O

Adding 600000 rows to HNSW Index

在milvus.yaml里，这些配置定义了读写并发数：
dataCoord.segmentCompactionPolicy.maxParallelReads = 8
dataCoord.segmentCompactionPolicy.maxParallelUploads = 4
另外还有：
dataNode.ioConcurrency = 0 ---------- 0意味着按照CPU×2 = 64 并发 I/O
dataNode.compaction.slotCap = 16 ------- 16个并发compation任务

再加上MinIO、datanode、indexnode 全塞在4 台物理机上，MinIO 与计算节点抢同一份 CPU/内存/网络，很容易造成I/O阻塞导致超时。

调整方向：

• 调大minio.requestTimeoutMs
• 把dataCoord.segmentCompactionPolicy.maxParallelReads/maxParallelUploads，dataNode.ioConcurrency，dataNode.compaction.slotCap这些调小
• 如果可能的话，compaction的时候不要带索引。等compaction完成后再重建索引

[eyeeye]

感谢大佬解答，排查下来发现确实是这个问题，集群配置了NodeLocalDNS缓解了这个问题， 另外今天发现10G的seg构建完hnsw索引，2176维度向量。索引文件大小是20G；我collection的源文件是8个segment文件，整个表就80G的大小；加载到queryNode的时候mixCoord内存校验没过,本以为很吃内存，调整了参数把全部内存都给queryNode,以为会占用160G内存，但是加载成功后发现4个queryNode总共也只用了80G的内存，这是为什么？
另外想咨询一下一个表建议是在多少个segment比较合适？我们目前线上大约会有500G的数据，想测试出性价比的机器配置是怎么样。实验环境试了多segment文件的读放大很影响性能,实验的数据是100个segment search top50性能很差，优化到25个seg提升很多但是依然不达标，是不是单segment性能最好？这个文件大小有计算公式参考吗，或者说主要还是看机器配置。

[yhmo]

感谢大佬解答，排查下来发现确实是这个问题，集群配置了NodeLocalDNS缓解了这个问题， 另外今天发现10G的seg构建完hnsw索引，2176维度向量。索引文件大小是20G；我collection的源文件是8个segment文件，整个表就80G的大小；加载到queryNode的时候mixCoord内存校验没过,本以为很吃内存，调整了参数把全部内存都给queryNode,以为会占用160G内存，但是加载成功后发现4个queryNode总共也只用了80G的内存，这是为什么？ 另外想咨询一下一个表建议是在多少个segment比较合适？我们目前线上大约会有500G的数据，想测试出性价比的机器配置是怎么样。实验环境试了多segment文件的读放大很影响性能,实验的数据是100个segment search top50性能很差，优化到25个seg提升很多但是依然不达标，是不是单segment性能最好？这个文件大小有计算公式参考吗，或者说主要还是看机器配置。

segment的parquet文件开启了压缩，所以会比真实储存的数据量小。你可以用pymilvus的list_persistent_segments这个接口列举一下各个segment的信息，看一下行数是多少，每条2176向量的大小大约是8KB，行数乘以8KB是真实大小。每个索引20GB才是接近真实的大小，因为新的存储版本里索引似乎没开启压缩。在旧版本里hnsw索引的大小开启了压缩，所以跟segment的大小接近。“queryNode总共也只用了80G的内存”-----这个有点反直觉，8个20GB的hnsw索引我觉得应该需要160GB内存。先确认每个segment的行数，然后检查mmap是否enabled

大型segment的优点是可以获得更高的性能，因为减少了很多reduce和调度的成本。缺点是同等数据量下内存峰值会高很多，资源配置得有更多的余量防止OOM。另外就是segment数量变少意味着数据只能分布在少量节点上，节点间不太好平衡(比如在3个节点，8个segment和16个segment的不平衡程度是不一样的)。
小型segment的优点是易于分布到多个节点上，内存峰值低资源利用率比较高。缺点就是调度成本高，性能差一些。
单个segment只是理论上性能最好，但实际和各种并行线程设置以及底层算法相关，未必能跑出最优的效果。对资源的需求也高，一个500GB的索引文件的二进制数据先分块并行读取进内存，然后在内存中构建图，内存校验要求2倍内存也就是1TB。而一个10GB的索引文件的内存校验只要求20GB。

[eyeeye]

当前的情况是80G的集合加载到2个queryNode上（8个segment *10G源文件，hnsw索引-20G大小一个），查看queryNode日志，发现加载失败，最后一个segment一直OOM显示内存不够。扩容queryNode到4个，加载成功。看attu每个node只占用了20G内存，服务上 free 命令查看，发现也只占用了20G内存，这一点有点反常识。milvus是否有一些机制是在加载的时候需要比较大的内存，但是加载到内存后，会有有一些优化机制导致加载成功之后内容占用是较少的。

[eyeeye]

发现是hiqueryNode.loadMemoryUsageFactor这个配置 默认设置是2

[yhmo]

8个segment *10G源文件 ---------------------- 在新版的storage里，segment的文件会被压缩，hnsw索引文件未压缩，所以这里的10GB segment实际上含有20GB的真实数据，根据collection的行数和向量维度估计即可看出来。

hnsw所以 20GB ------------------- 索引加载进querynode需要20GB内存，4个就是80GB。queryNode.loadMemoryUsageFactor指的是“加载索引时的内存准入安全系数”，因为加载20GB的内容时，内存的峰值远高于20GB，所以有个2倍安全系数。如果querynode的内存不够这个安全系数(也就是20GB42=160GB)，querynode就拒绝加载。如果改小的话就会有OOM的风险(改成1.5也可能成功)。

node实际占用20GB内存 -------------------- 有可能是开启了mmap