Milvus crashes with "fail to update resolver state" after configuring external S3 with custom CA

[gstefanoff]

Hi everyone,

I deployed a standalone instance of Milvus with helm chart (v5.0.3) with S3-compatible object storage.

I ran into the issue discussed here #26652 and tried the accepted answer. Deployment is now crashing with errors. If I disable externalS3.useSSL then pod fails with x509 errors.

Has anyone seen this error before when using external S3 with custom CA?

pod logs errors

[2026/07/23 21:32:16.564 +00:00] [WARN] [resolver/watch_based_grpc_resolver.go:54] ["fail to update resolver state"] [component=grpc-resolver] [scheme=milvus-session] [role=by-dev/meta/session/streamingnode] [exclusive=false] [id=3] [state="{"Addresses":[],"Endpoints":null,"ServiceConfig":null,"Attributes":null}"] [error="bad resolver state"]

[2026/07/23 21:32:16.564 +00:00] [WARN] [grpcclient/client.go:262] ["failed to get client address"] [error="find no available mixcoord, check mixcoord state"] [errorVerbose="find no available mixcoord, check mixcoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/distributed/mixcoord/client.(*Client).getCompatibleMixCoordAddr\n | \t/workspace/source/internal/distributed/mixcoord/client/client.go:145\n | github.com/milvus-io/milvus/internal/distributed/mixcoord/client.(*Client).getMixCoordAddr\n | \t/workspace/source/internal/distributed/mixcoord/client/client.go:120\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).connect\n | \t/workspace/source/internal/util/grpcclient/client.go:260\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).GetGrpcClient\n | \t/workspace/source/internal/util/grpcclient/client.go:226\n | github.com/milvus-io/milvus/internal/util/grpcclient.(

[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_2] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_3] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_7] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_1] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_14] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_5] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_12] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_10] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_4] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_0] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_8] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_6] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_9] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_13] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [INFO] [snmanager/streaming_node_manager.go:177] ["streaming node manager updated"] [assignments={}]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_15] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [WARN] [balancer/balancer_impl.go:500] ["channel is not assigned to any server"] [channel=by-dev-rootcoord-dml_11] [term=1] [serverID=0] [state=PCHANNEL_META_STATE_UNINITIALIZED]
[2026/07/23 21:32:16.582 +00:00] [INFO] [rootcoord/dml_channels.go:205] ["streaming service

] [WARN] [balancer/balancer_impl.go:261] ["fail to apply balance, start a backoff..."] [module=streamingcoord] [component=balancer] [policy=vchannelFair] [error="fail to balance: no available streaming node"] [errorVerbose="fail to balance: no available streaming node\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/streamingcoord/server/balancer.(*balancerImpl).balance\n | \t/workspace/source/internal/streamingcoord/server/balancer/balancer_impl.go:404\n | [...repeated from below...]\nWraps: (2) fail to balance\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/streamingcoord/server/balancer/policy/vchannelfair.(*policy).Balance\n | \t/workspace/source/internal/streamingcoord/server/balancer/policy/vchannelfair/vchannel_fair_policy.go:33\n | github.com/milvus-io/milvus/internal/streamingcoord/server/balancer.(*balancerImpl).balance\n | \t/workspace/source/internal/streamingcoord/server/balancer/balancer_impl.go:402\n | github.com/milvus-io/milvus/internal/streamingcoord/server/balancer.(*balancerImpl).balanceUntilNoChanged\n | \t/workspace/source/internal/streamingcoord/server/balancer/balancer_impl.go:373\n | github.com/milvus-io/milvus/internal/streamingcoord/server/balancer.(*balancerImpl).execute\n | \t/workspace/source/internal/streamingcoord/server/balancer/balancer_impl.go:256\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1700\nWraps: (4) no available streaming node\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError"]

[yhmo]

We don't use self-signed tls for minio/s3. The user made it work by installing the self signed certs in /etc/ssl/certs directory of all pods: #26652 (reply in thread). You can try this approach.