milvus-v2.4.6集群模式下，datacoord突然就挂了，为啥

[jhyever]

@yhmo 麻烦大佬帮忙看下

[yhmo]

这组报错是datacoord在启动时去连接etcd更新信息时超时，多半是etcd没正常工作。

[yhmo]

datacoord和rootcoord都有相同的报错
{"level":"warn","ts":"2025-03-05T09:00:35.326Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c80fc0/my-release-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}

这是etcd-client调用etcd的接口超时，看起来它们压根就连接不上etcd server

[yhmo]

你读一下这个页面里 https://milvus.io/docs/deploy_etcd.md 的 "Configure etcd on K8s"，对比一下你的配置

[jhyever]

1、连不上etcd原因是啥呀，如下是我的配置，etcd写入可能会是磁盘io机器问题嘛
2、如下是values.yaml文件的写入数据： 用的都是默认的配置

[jhyever]

rootcoord可以了，但是datacoord还是起不来呢{"level":"warn","ts":"2025-03-05T10:17:07.773Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0018fa540/my-release-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
{"level":"warn","ts":"2025-03-05T10:17:07.773Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0018fa540/my-release-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
{"level":"warn","ts":"2025-03-05T10:17:07.773Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0018fa540/my-release-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
{"level":"warn","ts":"2025-03-05T10:17:07.774Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0018fa540/my-release-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2025/03/05 10:17:07.774 +00:00] [WARN] [datacoord/server.go:687] ["retry func failed"] [retried=0] [error="context deadline exceeded"]
{"level":"warn","ts":"2025-03-05T10:17:17.975Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0018fa540/my-release-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2025/03/05 10:17:17.975 +00:00] [ERROR] [datacoord/kv_catalog.go:585] ["list index meta fail"] [prefix=field-index] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/metastore/kv/datacoord.(*Catalog).ListIndexes\n\t/workspace/source/internal/metastore/kv/datacoord/kv_catalog.go:585\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexMeta).reloadFromKV\n\t/workspace/source/internal/datacoord/index_meta.go:78\ngithub.com/milvus-io/milvus/internal/datacoord.newIndexMeta\n\t/workspace/source/internal/datacoord/index_meta.go:67\ngithub.com/milvus-io/milvus/internal/datacoord.newMeta\n\t/workspace/source/internal/datacoord/meta.go:96\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1\n\t/workspace/source/internal/datacoord/server.go:673\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/workspace/source/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta\n\t/workspace/source/internal/datacoord/server.go:687\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/workspace/source/internal/datacoord/server.go:354\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/workspace/source/internal/datacoord/server.go:334\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/workspace/source/internal/distributed/datacoord/service.go:129\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/workspace/source/internal/distributed/datacoord/service.go:256\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/workspace/source/cmd/components/data_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"]
[2025/03/05 10:17:17.975 +00:00] [ERROR] [datacoord/index_meta.go:80] ["indexMeta reloadFromKV load field indexes fail"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*indexMeta).reloadFromKV\n\t/workspace/source/internal/datacoord/index_meta.go:80\ngithub.com/milvus-io/milvus/internal/datacoord.newIndexMeta\n\t/workspace/source/internal/datacoord/index_meta.go:67\ngithub.com/milvus-io/milvus/internal/datacoord.newMeta\n\t/workspace/source/internal/datacoord/meta.go:96\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1\n\t/workspace/source/internal/datacoord/server.go:673\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/workspace/source/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta\n\t/workspace/source/internal/datacoord/server.go:687\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/workspace/source/internal/datacoord/server.go:354\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/workspace/source/internal/datacoord/server.go:334\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/workspace/source/internal/distributed/datacoord/service.go:129\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/workspace/source/internal/distributed/datacoord/service.go:256\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/workspace/source/cmd/components/data_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"]
{"level":"warn","ts":"2025-03-05T10:17:28.376Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0018fa540/my-release-etcd:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2025/03/05 10:17:28.376 +00:00] [ERROR] [datacoord/kv_catalog.go:585] ["list index meta fail"] [prefix=field-index] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/metastore/kv/datacoord.(*Catalog).ListIndexes\n\t/workspace/source/internal/metastore/kv/datacoord/kv_catalog.go:585\ngithub.com/milvus-io/milvus/internal/datacoord.(*indexMeta).reloadFromKV\n\t/workspace/source/internal/datacoord/index_meta.go:78\ngithub.com/milvus-io/milvus/internal/datacoord.newIndexMeta\n\t/workspace/source/internal/datacoord/index_meta.go:67\ngithub.com/milvus-io/milvus/internal/datacoord.newMeta\n\t/workspace/source/internal/datacoord/meta.go:96\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1\n\t/workspace/source/internal/datacoord/server.go:673\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/workspace/source/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta\n\t/workspace/source/internal/datacoord/server.go:687\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/workspace/source/internal/datacoord/server.go:354\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/workspace/source/internal/datacoord/server.go:334\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/workspace/source/internal/distributed/datacoord/service.go:129\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/workspace/source/internal/distributed/datacoord/service.go:256\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/workspace/source/cmd/components/data_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"]
[2025/03/05 10:17:28.376 +00:00] [ERROR] [datacoord/index_meta.go:80] ["indexMeta reloadFromKV load field indexes fail"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*indexMeta).reloadFromKV\n\t/workspace/source/internal/datacoord/index_meta.go:80\ngithub.com/milvus-io/milvus/internal/datacoord.newIndexMeta\n\t/workspace/source/internal/datacoord/index_meta.go:67\ngithub.com/milvus-io/milvus/internal/datacoord.newMeta\n\t/workspace/source/internal/datacoord/meta.go:96\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1\n\t/workspace/source/internal/datacoord/server.go:673\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/workspace/source/pkg/util/retry/retry.go:44\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta\n\t/workspace/source/internal/datacoord/server.go:687\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initDataCoord\n\t/workspace/source/internal/datacoord/server.go:354\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/workspace/source/internal/datacoord/server.go:334\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/workspace/source/internal/distributed/datacoord/service.go:129\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/workspace/source/internal/distributed/datacoord/service.go:256\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/workspace/source/cmd/components/data_coord.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:114"]

[yhmo]

报错是etcd client层面的，已经不是milvus的代码问题，也不是milvus功能问题。
估计节点间的网络有问题，这种问题我们也没法解决。