Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: panic: Endpoint url cannot have fully qualified paths. #40471

Open
1 task done
zyyworktable opened this issue Mar 7, 2025 · 4 comments
Open
1 task done

[Bug]: panic: Endpoint url cannot have fully qualified paths. #40471

zyyworktable opened this issue Mar 7, 2025 · 4 comments
Assignees
@zyyworktable
Labels
area/kubernetes help wanted Extra attention is needed kind/bug Issues or changes related a bug

Comments

@zyyworktable
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:v2.5.5
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):     rocksmq

Current Behavior

We attempted to deploy the Docker Compose version of milvus-standalone in a Kubernetes cluster by translating the docker-compose.yaml file into Kubernetes YAML configuration files. The Docker Compose setup for milvus-standalone requires spinning up three containers: etcd, minio, and milvusdb. While we successfully launched the etcd and minio containers, we encountered an issue when starting the milvusdb image, which displayed the following error:
我们尝试在k8s集群中部署 milvus-standalone 的docker compose版本，具体通过根据docker-compose.yaml编写 k8s的yaml配置文件。milvus-standalone 的docker compose版本 共需要拉起三个容器：etcd、minio和milvsdb，我们已经顺利拉起了etcd、minio 容器，但是在milvusdb镜像启动时遇到了问题，显示报错如下：
2025-03-07T17:09:56.777899664+08:00 ERROR: ld.so: object '/milvus/lib/' from LD_PRELOAD cannot be preloaded (cannot read file data): ignored.

2025-03-07T17:09:56.926627151+08:00 [2025/03/07 09:09:56.926 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"]

2025-03-07T17:09:56.938125501+08:00 2025/03/07 09:09:56 maxprocs: Updating GOMAXPROCS=16: determined from CPU quota

2025-03-07T17:09:56.938317336+08:00

2025-03-07T17:09:56.938319556+08:00 __ _________ _ ____ ______

2025-03-07T17:09:56.938320909+08:00 / |/ / _/ /| | / / / / / __/

2025-03-07T17:09:56.938322312+08:00 / /|/ // // /| |/ / /_/ /\ \

2025-03-07T17:09:56.938323691+08:00 // /////_/__/

2025-03-07T17:09:56.938324875+08:00

2025-03-07T17:09:56.938326532+08:00 Welcome to use Milvus!

2025-03-07T17:09:56.938327991+08:00 Version: pkg/v2.5.5

2025-03-07T17:09:56.938329309+08:00 Built: Wed Feb 26 04:01:36 UTC 2025

2025-03-07T17:09:56.938330611+08:00 GitCommit: 27fb8d9

2025-03-07T17:09:56.938331814+08:00 GoVersion: go version go1.22.0 linux/amd64

2025-03-07T17:09:56.938332916+08:00

2025-03-07T17:09:56.939287421+08:00 TotalMem: 68719476736

2025-03-07T17:09:56.939304112+08:00 UsedMem: 97447936

2025-03-07T17:09:56.939306540+08:00

2025-03-07T17:09:56.939315299+08:00 open pid file: /run/milvus/standalone.pid

2025-03-07T17:09:56.939316991+08:00 lock pid file: /run/milvus/standalone.pid

2025-03-07T17:09:56.939634057+08:00 [2025/03/07 09:09:56.939 +00:00] [INFO] [roles/roles.go:344] ["starting running Milvus components"]

2025-03-07T17:09:56.939643360+08:00 [2025/03/07 09:09:56.939 +00:00] [INFO] [roles/roles.go:186] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/:/milvus/lib/libjemalloc.so]

2025-03-07T17:09:56.944326589+08:00 [2025/03/07 09:09:56.944 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]

2025-03-07T17:09:56.945074083+08:00 [2025/03/07 09:09:56.944 +00:00] [DEBUG] [paramtable/base_table.go:213] ["init etcd source"] [etcdInfo="{"UseEmbed":false,"EnableAuth":false,"UserName":"","PassWord":"","UseSSL":false,"Endpoints":["http://****:2379"],"KeyPrefix":"by-dev","CertFile":"/path/to/etcd-client.pem","KeyFile":"/path/to/etcd-client-key.pem","CaCertFile":"/path/to/ca.pem","MinVersion":"1.3","RefreshInterval":5000000000}"]

2025-03-07T17:09:56.945088846+08:00 [2025/03/07 09:09:56.945 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://****:2379]"] [minVersion=1.3]

2025-03-07T17:09:56.946253798+08:00 [2025/03/07 09:09:56.946 +00:00] [DEBUG] [config/etcd_source.go:92] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[http://****:2379]"]

2025-03-07T17:09:56.946317428+08:00 [2025/03/07 09:09:56.946 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=EtcdSource]

2025-03-07T17:09:56.948684990+08:00 [2025/03/07 09:09:56.948 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]

2025-03-07T17:09:56.948687296+08:00 [2025/03/07 09:09:56.948 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]

2025-03-07T17:09:56.948772528+08:00 [2025/03/07 09:09:56.948 +00:00] [INFO] [tracer/tracer.go:50] ["Init tracer finished"] [Exporter=noop]

2025-03-07T17:09:56.948777714+08:00 [2025/03/07 09:09:56.948 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://*****:2379]"] [minVersion=1.3]

2025-03-07T17:09:56.949698285+08:00 [2025/03/07 09:09:56.949 +00:00] [INFO] [resolver/resolver_with_discoverer.go:128] ["service discover task started, listening..."] [scheme=milvus-session]

2025-03-07T17:09:56.950215598+08:00 [2025/03/07 09:09:56.949 +00:00] [INFO] [resolver/resolver_with_discoverer.go:160] ["service discover update, update resolver"] [scheme=milvus-session] [state="{"Version":40,"State":{"Addresses":[],"Endpoints":null,"ServiceConfig":null,"Attributes":null}}"] [resolver_count=0]

2025-03-07T17:09:56.950220093+08:00 [2025/03/07 09:09:56.950 +00:00] [INFO] [resolver/resolver_with_discoverer.go:170] ["update resolver done"] [scheme=milvus-session]

2025-03-07T17:09:56.954434775+08:00 [2025/03/07 09:09:56.954 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]

2025-03-07T17:09:56.954437713+08:00 [2025/03/07 09:09:56.954 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]

2025-03-07T17:09:56.954490173+08:00 [2025/03/07 09:09:56.954 +00:00] [INFO] [roles/roles.go:293] [setupPrometheusHTTPServer]

2025-03-07T17:09:56.954555478+08:00 [2025/03/07 09:09:56.954 +00:00] [INFO] [http/server.go:240] ["management listen"] [addr=:9091]

2025-03-07T17:09:56.954564149+08:00 [2025/03/07 09:09:56.954 +00:00] [WARN] [resolver/watch_based_grpc_resolver.go:55] ["fail to update resolver state"] [scheme=milvus-session] [id=1] [error="bad resolver state"]

2025-03-07T17:09:56.955151019+08:00 [2025/03/07 09:09:56.955 +00:00] [INFO] [gc/gc_tuner.go:138] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=61847529062]

2025-03-07T17:09:56.955211888+08:00 [2025/03/07 09:09:56.955 +00:00] [INFO] [rootcoord/root_coord.go:165] ["update rootcoord state"] [state=Abnormal]

2025-03-07T17:09:56.955315135+08:00 [2025/03/07 09:09:56.955 +00:00] [INFO] [rootcoord/service.go:155] ["RootCoord listen on"] [address="[::]:53100"] [port=53100]

2025-03-07T17:09:56.955320543+08:00 [2025/03/07 09:09:56.955 +00:00] [INFO] [rootcoord/service.go:185] ["init params done.."]

2025-03-07T17:09:56.955327175+08:00 [2025/03/07 09:09:56.955 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://****:2379]"] [minVersion=1.3]

2025-03-07T17:09:56.956124892+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [rootcoord/service.go:205] ["etcd connect done ..."]

2025-03-07T17:09:56.956128003+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [rootcoord/service.go:219] ["RootCoord start to create DataCoord client"]

2025-03-07T17:09:56.956520422+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [rootcoord/service.go:228] ["RootCoord start to create QueryCoord client"]

2025-03-07T17:09:56.956765130+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [rootcoord/root_coord.go:524] ["register metrics actions finished"]

2025-03-07T17:09:56.956799559+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]

2025-03-07T17:09:56.956801088+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [datacoord/service.go:97] ["DataCoord listen on"] [address="[::]:13333"] [port=13333]

2025-03-07T17:09:56.956830624+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://*****:2379]"] [minVersion=1.3]

2025-03-07T17:09:56.957176730+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [components/index_coord.go:42] ["IndexCoord running ..."]

2025-03-07T17:09:56.957222867+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [querycoordv2/server.go:674] ["update querycoord state"] [state=Abnormal]

2025-03-07T17:09:56.957314654+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [querycoord/service.go:104] ["QueryCoord listen on"] [address="[::]:19531"] [port=19531]

2025-03-07T17:09:56.957332899+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://*****:2379]"] [minVersion=1.3]

2025-03-07T17:09:56.957704864+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [datacoord/server.go:1206] ["register metrics actions finished"]

2025-03-07T17:09:56.957727315+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]

2025-03-07T17:09:56.959494767+08:00 [2025/03/07 09:09:56.959 +00:00] [INFO] [querynode/service.go:102] ["QueryNode listen on"] [address="[::]:21123"] [port=21123]

2025-03-07T17:09:56.959516471+08:00 [2025/03/07 09:09:56.959 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://****:2379]"] [minVersion=1.3]

2025-03-07T17:09:56.960756087+08:00 [2025/03/07 09:09:56.960 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]

2025-03-07T17:09:56.960899626+08:00 [2025/03/07 09:09:56.960 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]

2025-03-07T17:09:56.960990561+08:00 [2025/03/07 09:09:56.960 +00:00] [INFO] [coordclient/registry.go:63] ["register query coord server"] [enableLocalClient="{"ServerType":"standalone","EnableQueryCoord":true,"EnableDataCoord":true,"EnableRootCoord":true}"]

2025-03-07T17:09:56.961459348+08:00 [2025/03/07 09:09:56.961 +00:00] [INFO] [datanode/service.go:102] ["DataNode listen on"] [address="[::]:21124"] [port=21124]

2025-03-07T17:09:56.961477343+08:00 [2025/03/07 09:09:56.961 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://*****:2379]"] [minVersion=1.3]

2025-03-07T17:09:56.961901411+08:00 [2025/03/07 09:09:56.961 +00:00] [INFO] [indexnode/service.go:78] ["IndexNode listen on"] [address="[::]:21121"] [port=21121]

2025-03-07T17:09:56.961942857+08:00 [2025/03/07 09:09:56.961 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]

2025-03-07T17:09:56.962083237+08:00 [2025/03/07 09:09:56.962 +00:00] [INFO] [datanode/service.go:257] ["DataNode address"] [address=****:21124]

2025-03-07T17:09:56.962195182+08:00 [2025/03/07 09:09:56.962 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]

2025-03-07T17:09:56.962211032+08:00 [2025/03/07 09:09:56.962 +00:00] [INFO] [proxy/look_aside_balancer.go:233] ["Start check query node health loop"]

2025-03-07T17:09:56.962339833+08:00 [2025/03/07 09:09:56.962 +00:00] [INFO] [hookutil/hook.go:70] ["empty so path, skip to load plugin"]

2025-03-07T17:09:56.962415014+08:00 [2025/03/07 09:09:56.962 +00:00] [INFO] [vecindexmgr/vector_index_mgr.go:124] ["init vector indexes with features : BINFLAT : 720897,BIN_FLAT : 720897,BIN_IVF_FLAT : 524289,DISKANN : 2097166,FLAT : 720942,HNSW : 1572910,HNSWLIB_DEPRECATED : 1572879,HNSW_PQ : 1572910,HNSW_PRQ : 1572910,HNSW_SQ : 1572910,IVFBIN : 524289,IVFFLAT : 524334,IVFFLATCC : 46,IVFPQ : 524334,IVFSQ : 524334,IVF_FLAT : 524334,IVF_FLAT_CC : 46,IVF_PQ : 524334,IVF_SQ : 524334,IVF_SQ8 : 524334,IVF_SQ_CC : 46,SCANN : 524334,SCANN_DVR : 14,SPARSE_INVERTED_INDEX : 524304,SPARSE_INVERTED_INDEX_CC : 524304,SPARSE_WAND : 524304,SPARSE_WAND_CC : 524304,"]

2025-03-07T17:09:56.963349922+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/listener_manager.go:53] ["Proxy listen on external grpc listener"] [address=****:19530] [port=19530]

2025-03-07T17:09:56.963412132+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/listener_manager.go:63] ["Proxy listen on internal grpc listener"] [address=*****:19529] [port=19529]

2025-03-07T17:09:56.963424098+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/listener_manager.go:98] ["Proxy server(http) and external grpc server share the same port"]

2025-03-07T17:09:56.963430183+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/service.go:410] ["init Proxy server"]

2025-03-07T17:09:56.963447258+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/service.go:430] ["Proxy init service's parameter table done"]

2025-03-07T17:09:56.963455476+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/service.go:432] ["Proxy init http server's parameter table done"]

2025-03-07T17:09:56.963484780+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [accesslog/global.go:146] ["Init access logger success"]

2025-03-07T17:09:56.963496759+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/service.go:436] ["init Proxy's tracer done"] ["service name"="Proxy ip: ******, port: 19530"]

2025-03-07T17:09:56.963518653+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://******:2379]"] [minVersion=1.3]

2025-03-07T17:09:56.964166605+08:00 [2025/03/07 09:09:56.964 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]

2025-03-07T17:09:56.964256329+08:00 [2025/03/07 09:09:56.964 +00:00] [INFO] [proxy/service.go:386] ["create Proxy internal grpc server"] ["enforcement policy"="{"MinTime":5000000000,"PermitWithoutStream":true}"] ["server parameters"="{"MaxConnectionIdle":0,"MaxConnectionAge":0,"MaxConnectionAgeGrace":0,"Time":60000000000,"Timeout":10000000000}"]

2025-03-07T17:09:56.964832355+08:00 [2025/03/07 09:09:56.964 +00:00] [WARN] [client/client.go:104] ["RootCoordClient mess key not exist"] [key=rootcoord]

2025-03-07T17:09:56.964835121+08:00 [2025/03/07 09:09:56.964 +00:00] [WARN] [grpcclient/client.go:262] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]

2025-03-07T17:09:56.964839954+08:00 [2025/03/07 09:09:56.964 +00:00] [WARN] [grpcclient/client.go:473] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]

2025-03-07T17:09:56.964964256+08:00 [2025/03/07 09:09:56.964 +00:00] [WARN] [grpcclient/client.go:494] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:493\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:128\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:486\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:589\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:135\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n | \t/workspace/source/internal/distributed/proxy/service.go:485\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n | \t/workspace/source/internal/distributed/proxy/service.go:411\n | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n | \t/workspace/source/cmd/components/proxy.go:60\n | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n | \t/workspace/source/cmd/roles/roles.go:129\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]

2025-03-07T17:09:56.965158175+08:00 [2025/03/07 09:09:56.965 +00:00] [WARN] [client/client.go:104] ["RootCoordClient mess key not exist"] [key=rootcoord]

2025-03-07T17:09:56.965160056+08:00 [2025/03/07 09:09:56.965 +00:00] [WARN] [grpcclient/client.go:262] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]

2025-03-07T17:09:56.965165399+08:00 [2025/03/07 09:09:56.965 +00:00] [WARN] [grpcclient/client.go:480] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]

2025-03-07T17:09:56.965242031+08:00 [2025/03/07 09:09:56.965 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:493\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:128\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:486\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:589\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:135\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n | github.com/milvus-io/milvus/pkg/v2/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n | \t/workspace/source/internal/distributed/proxy/service.go:485\n | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n | \t/workspace/source/internal/distributed/proxy/service.go:411\n | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n | \t/workspace/source/cmd/components/proxy.go:60\n | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n | \t/workspace/source/cmd/roles/roles.go:129\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]

2025-03-07T17:09:57.060658134+08:00 [2025/03/07 09:09:57.060 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]

2025-03-07T17:09:57.060687797+08:00 [2025/03/07 09:09:57.060 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]

2025-03-07T17:09:57.060967179+08:00 [2025/03/07 09:09:57.060 +00:00] [INFO] [querynodev2/server.go:304] ["register metrics actions finished"]

2025-03-07T17:09:57.060972958+08:00 [2025/03/07 09:09:57.060 +00:00] [INFO] [querynodev2/server.go:314] ["QueryNode session info"] [metaPath=by-dev/meta]

2025-03-07T17:09:57.062820549+08:00 [2025/03/07 09:09:57.062 +00:00] [INFO] [datanode/service.go:266] ["initializing RootCoord client for DataNode"]

2025-03-07T17:09:57.062826516+08:00 [2025/03/07 09:09:57.062 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://******:2379]"] [minVersion=1.3]

2025-03-07T17:09:57.063363628+08:00 [2025/03/07 09:09:57.063 +00:00] [INFO] [indexnode/indexnode.go:231] ["IndexNode init"] [state=Initializing]

2025-03-07T17:09:57.064412355+08:00 [2025/03/07 09:09:57.064 +00:00] [INFO] [querycoordv2/server.go:252] ["QueryCoord start init"] [meta-root-path=by-dev/meta] [address=*****:19531]

2025-03-07T17:09:57.064429332+08:00 [2025/03/07 09:09:57.064 +00:00] [INFO] [querycoordv2/server.go:247] ["register metrics actions finished"]

2025-03-07T17:09:57.065601657+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [sessionutil/session_util.go:305] ["start server"] [name=indexcoord] [address=****:13333] [id=21] [server_labels={}]

2025-03-07T17:09:57.065666269+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [sessionutil/session_util.go:305] ["start server"] [name=querynode] [address=****:21123] [id=21] [server_labels={}]

2025-03-07T17:09:57.065729070+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [sessionutil/session_util.go:305] ["start server"] [name=rootcoord] [address=****:53100] [id=21] [server_labels={}]

2025-03-07T17:09:57.065793899+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [sessionutil/session_util.go:1273] ["save server info into file"] [content="querynode-21\n"] [filePath=/tmp/milvus/server_id_1]

2025-03-07T17:09:57.065795963+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [querynodev2/server.go:171] ["QueryNode init session"] [nodeID=21] ["node address"=*****:21123]

2025-03-07T17:09:57.065832971+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [resolver/resolver_with_discoverer.go:128] ["service discover task started, listening..."] [scheme=milvus-session]

2025-03-07T17:09:57.065855941+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]

2025-03-07T17:09:57.065857829+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [rootcoord/root_coord.go:165] ["update rootcoord state"] [state=Initializing]

2025-03-07T17:09:57.065859520+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]

2025-03-07T17:09:57.065968131+08:00 [2025/03/07 09:09:57.065 +00:00] [ERROR] [querynodev2/server.go:345] ["QueryNode init vector storage failed"] [error="Endpoint url cannot have fully qualified paths."] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:345\nsync.(*Once).doSlow\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/sync/once.go:74\nsync.(*Once).Do\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:311\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:144\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:227\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:59\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:129"]

2025-03-07T17:09:57.065993317+08:00 [2025/03/07 09:09:57.065 +00:00] [ERROR] [querynode/service.go:145] ["QueryNode init error: "] [error="Endpoint url cannot have fully qualified paths."] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:145\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:227\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:59\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:129"]

2025-03-07T17:09:57.066000276+08:00 [2025/03/07 09:09:57.065 +00:00] [ERROR] [components/query_node.go:60] ["QueryNode starts error"] [error="Endpoint url cannot have fully qualified paths."] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:60\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:129"]

2025-03-07T17:09:57.069184984+08:00 panic: Endpoint url cannot have fully qualified paths.

Expected Behavior

No response

Steps To Reproduce



Milvus Log


2025-03-07T17:09:56.777899664+08:00 ERROR: ld.so: object '/milvus/lib/' from LD_PRELOAD cannot be preloaded (cannot read file data): ignored.


2025-03-07T17:09:56.926627151+08:00 [2025/03/07 09:09:56.926 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"]


2025-03-07T17:09:56.938125501+08:00 2025/03/07 09:09:56 maxprocs: Updating GOMAXPROCS=16: determined from CPU quota


2025-03-07T17:09:56.938317336+08:00


2025-03-07T17:09:56.938319556+08:00     __  _________ _   ____  ______


2025-03-07T17:09:56.938320909+08:00    /  |/  /  _/ /| | / / / / / __/


2025-03-07T17:09:56.938322312+08:00   / /|/ // // /| |/ / /_/ /\ \


2025-03-07T17:09:56.938323691+08:00  //  /////_/__/


2025-03-07T17:09:56.938324875+08:00


2025-03-07T17:09:56.938326532+08:00 Welcome to use Milvus!


2025-03-07T17:09:56.938327991+08:00 Version:   pkg/v2.5.5


2025-03-07T17:09:56.938329309+08:00 Built:     Wed Feb 26 04:01:36 UTC 2025


2025-03-07T17:09:56.938330611+08:00 GitCommit: 27fb8d9


2025-03-07T17:09:56.938331814+08:00 GoVersion: go version go1.22.0 linux/amd64


2025-03-07T17:09:56.938332916+08:00


2025-03-07T17:09:56.939287421+08:00 TotalMem: 68719476736


2025-03-07T17:09:56.939304112+08:00 UsedMem: 97447936


2025-03-07T17:09:56.939306540+08:00


2025-03-07T17:09:56.939315299+08:00 open pid file: /run/milvus/standalone.pid


2025-03-07T17:09:56.939316991+08:00 lock pid file: /run/milvus/standalone.pid


2025-03-07T17:09:56.939634057+08:00 [2025/03/07 09:09:56.939 +00:00] [INFO] [roles/roles.go:344] ["starting running Milvus components"]


2025-03-07T17:09:56.939643360+08:00 [2025/03/07 09:09:56.939 +00:00] [INFO] [roles/roles.go:186] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/:/milvus/lib/libjemalloc.so]


2025-03-07T17:09:56.944326589+08:00 [2025/03/07 09:09:56.944 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]


2025-03-07T17:09:56.945074083+08:00 [2025/03/07 09:09:56.944 +00:00] [DEBUG] [paramtable/base_table.go:213] ["init etcd source"] [etcdInfo="{"UseEmbed":false,"EnableAuth":false,"UserName":"","PassWord":"","UseSSL":false,"Endpoints":["http://****:2379"],"KeyPrefix":"by-dev","CertFile":"/path/to/etcd-client.pem","KeyFile":"/path/to/etcd-client-key.pem","CaCertFile":"/path/to/ca.pem","MinVersion":"1.3","RefreshInterval":5000000000}"]


2025-03-07T17:09:56.945088846+08:00 [2025/03/07 09:09:56.945 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://****:2379]"] [minVersion=1.3]


2025-03-07T17:09:56.946253798+08:00 [2025/03/07 09:09:56.946 +00:00] [DEBUG] [config/etcd_source.go:92] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[http://****:2379]"]


2025-03-07T17:09:56.946317428+08:00 [2025/03/07 09:09:56.946 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=EtcdSource]


2025-03-07T17:09:56.948684990+08:00 [2025/03/07 09:09:56.948 +00:00] [DEBUG] [runtime/asm_amd64.s:1695] ["start refreshing configurations"] [source=FileSource]


2025-03-07T17:09:56.948687296+08:00 [2025/03/07 09:09:56.948 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]


2025-03-07T17:09:56.948772528+08:00 [2025/03/07 09:09:56.948 +00:00] [INFO] [tracer/tracer.go:50] ["Init tracer finished"] [Exporter=noop]


2025-03-07T17:09:56.948777714+08:00 [2025/03/07 09:09:56.948 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://*****:2379]"] [minVersion=1.3]


2025-03-07T17:09:56.949698285+08:00 [2025/03/07 09:09:56.949 +00:00] [INFO] [resolver/resolver_with_discoverer.go:128] ["service discover task started, listening..."] [scheme=milvus-session]


2025-03-07T17:09:56.950215598+08:00 [2025/03/07 09:09:56.949 +00:00] [INFO] [resolver/resolver_with_discoverer.go:160] ["service discover update, update resolver"] [scheme=milvus-session] [state="{"Version":40,"State":{"Addresses":[],"Endpoints":null,"ServiceConfig":null,"Attributes":null}}"] [resolver_count=0]


2025-03-07T17:09:56.950220093+08:00 [2025/03/07 09:09:56.950 +00:00] [INFO] [resolver/resolver_with_discoverer.go:170] ["update resolver done"] [scheme=milvus-session]


2025-03-07T17:09:56.954434775+08:00 [2025/03/07 09:09:56.954 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]


2025-03-07T17:09:56.954437713+08:00 [2025/03/07 09:09:56.954 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]


2025-03-07T17:09:56.954490173+08:00 [2025/03/07 09:09:56.954 +00:00] [INFO] [roles/roles.go:293] [setupPrometheusHTTPServer]


2025-03-07T17:09:56.954555478+08:00 [2025/03/07 09:09:56.954 +00:00] [INFO] [http/server.go:240] ["management listen"] [addr=:9091]


2025-03-07T17:09:56.954564149+08:00 [2025/03/07 09:09:56.954 +00:00] [WARN] [resolver/watch_based_grpc_resolver.go:55] ["fail to update resolver state"] [scheme=milvus-session] [id=1] [error="bad resolver state"]


2025-03-07T17:09:56.955151019+08:00 [2025/03/07 09:09:56.955 +00:00] [INFO] [gc/gc_tuner.go:138] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=61847529062]


2025-03-07T17:09:56.955211888+08:00 [2025/03/07 09:09:56.955 +00:00] [INFO] [rootcoord/root_coord.go:165] ["update rootcoord state"] [state=Abnormal]


2025-03-07T17:09:56.955315135+08:00 [2025/03/07 09:09:56.955 +00:00] [INFO] [rootcoord/service.go:155] ["RootCoord listen on"] [address="[::]:53100"] [port=53100]


2025-03-07T17:09:56.955320543+08:00 [2025/03/07 09:09:56.955 +00:00] [INFO] [rootcoord/service.go:185] ["init params done.."]


2025-03-07T17:09:56.955327175+08:00 [2025/03/07 09:09:56.955 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://****:2379]"] [minVersion=1.3]


2025-03-07T17:09:56.956124892+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [rootcoord/service.go:205] ["etcd connect done ..."]


2025-03-07T17:09:56.956128003+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [rootcoord/service.go:219] ["RootCoord start to create DataCoord client"]


2025-03-07T17:09:56.956520422+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [rootcoord/service.go:228] ["RootCoord start to create QueryCoord client"]


2025-03-07T17:09:56.956765130+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [rootcoord/root_coord.go:524] ["register metrics actions finished"]


2025-03-07T17:09:56.956799559+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]


2025-03-07T17:09:56.956801088+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [datacoord/service.go:97] ["DataCoord listen on"] [address="[::]:13333"] [port=13333]


2025-03-07T17:09:56.956830624+08:00 [2025/03/07 09:09:56.956 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://*****:2379]"] [minVersion=1.3]


2025-03-07T17:09:56.957176730+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [components/index_coord.go:42] ["IndexCoord running ..."]


2025-03-07T17:09:56.957222867+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [querycoordv2/server.go:674] ["update querycoord state"] [state=Abnormal]


2025-03-07T17:09:56.957314654+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [querycoord/service.go:104] ["QueryCoord listen on"] [address="[::]:19531"] [port=19531]


2025-03-07T17:09:56.957332899+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://*****:2379]"] [minVersion=1.3]


2025-03-07T17:09:56.957704864+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [datacoord/server.go:1206] ["register metrics actions finished"]


2025-03-07T17:09:56.957727315+08:00 [2025/03/07 09:09:56.957 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]


2025-03-07T17:09:56.959494767+08:00 [2025/03/07 09:09:56.959 +00:00] [INFO] [querynode/service.go:102] ["QueryNode listen on"] [address="[::]:21123"] [port=21123]


2025-03-07T17:09:56.959516471+08:00 [2025/03/07 09:09:56.959 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://****:2379]"] [minVersion=1.3]


2025-03-07T17:09:56.960756087+08:00 [2025/03/07 09:09:56.960 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]


2025-03-07T17:09:56.960899626+08:00 [2025/03/07 09:09:56.960 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]


2025-03-07T17:09:56.960990561+08:00 [2025/03/07 09:09:56.960 +00:00] [INFO] [coordclient/registry.go:63] ["register query coord server"] [enableLocalClient="{"ServerType":"standalone","EnableQueryCoord":true,"EnableDataCoord":true,"EnableRootCoord":true}"]


2025-03-07T17:09:56.961459348+08:00 [2025/03/07 09:09:56.961 +00:00] [INFO] [datanode/service.go:102] ["DataNode listen on"] [address="[::]:21124"] [port=21124]


2025-03-07T17:09:56.961477343+08:00 [2025/03/07 09:09:56.961 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://*****:2379]"] [minVersion=1.3]


2025-03-07T17:09:56.961901411+08:00 [2025/03/07 09:09:56.961 +00:00] [INFO] [indexnode/service.go:78] ["IndexNode listen on"] [address="[::]:21121"] [port=21121]


2025-03-07T17:09:56.961942857+08:00 [2025/03/07 09:09:56.961 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]


2025-03-07T17:09:56.962083237+08:00 [2025/03/07 09:09:56.962 +00:00] [INFO] [datanode/service.go:257] ["DataNode address"] [address=****:21124]


2025-03-07T17:09:56.962195182+08:00 [2025/03/07 09:09:56.962 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]


2025-03-07T17:09:56.962211032+08:00 [2025/03/07 09:09:56.962 +00:00] [INFO] [proxy/look_aside_balancer.go:233] ["Start check query node health loop"]


2025-03-07T17:09:56.962339833+08:00 [2025/03/07 09:09:56.962 +00:00] [INFO] [hookutil/hook.go:70] ["empty so path, skip to load plugin"]


2025-03-07T17:09:56.962415014+08:00 [2025/03/07 09:09:56.962 +00:00] [INFO] [vecindexmgr/vector_index_mgr.go:124] ["init vector indexes with features : BINFLAT : 720897,BIN_FLAT : 720897,BIN_IVF_FLAT : 524289,DISKANN : 2097166,FLAT : 720942,HNSW : 1572910,HNSWLIB_DEPRECATED : 1572879,HNSW_PQ : 1572910,HNSW_PRQ : 1572910,HNSW_SQ : 1572910,IVFBIN : 524289,IVFFLAT : 524334,IVFFLATCC : 46,IVFPQ : 524334,IVFSQ : 524334,IVF_FLAT : 524334,IVF_FLAT_CC : 46,IVF_PQ : 524334,IVF_SQ : 524334,IVF_SQ8 : 524334,IVF_SQ_CC : 46,SCANN : 524334,SCANN_DVR : 14,SPARSE_INVERTED_INDEX : 524304,SPARSE_INVERTED_INDEX_CC : 524304,SPARSE_WAND : 524304,SPARSE_WAND_CC : 524304,"]


2025-03-07T17:09:56.963349922+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/listener_manager.go:53] ["Proxy listen on external grpc listener"] [address=****:19530] [port=19530]


2025-03-07T17:09:56.963412132+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/listener_manager.go:63] ["Proxy listen on internal grpc listener"] [address=*****:19529] [port=19529]


2025-03-07T17:09:56.963424098+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/listener_manager.go:98] ["Proxy server(http) and external grpc server share the same port"]


2025-03-07T17:09:56.963430183+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/service.go:410] ["init Proxy server"]


2025-03-07T17:09:56.963447258+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/service.go:430] ["Proxy init service's parameter table done"]


2025-03-07T17:09:56.963455476+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/service.go:432] ["Proxy init http server's parameter table done"]


2025-03-07T17:09:56.963484780+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [accesslog/global.go:146] ["Init access logger success"]


2025-03-07T17:09:56.963496759+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [proxy/service.go:436] ["init Proxy's tracer done"] ["service name"="Proxy ip: ******, port: 19530"]


2025-03-07T17:09:56.963518653+08:00 [2025/03/07 09:09:56.963 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://******:2379]"] [minVersion=1.3]


2025-03-07T17:09:56.964166605+08:00 [2025/03/07 09:09:56.964 +00:00] [INFO] [utils/util.go:60] ["Internal TLS Enabled"] [value=false]


2025-03-07T17:09:56.964256329+08:00 [2025/03/07 09:09:56.964 +00:00] [INFO] [proxy/service.go:386] ["create Proxy internal grpc server"] ["enforcement policy"="{"MinTime":5000000000,"PermitWithoutStream":true}"] ["server parameters"="{"MaxConnectionIdle":0,"MaxConnectionAge":0,"MaxConnectionAgeGrace":0,"Time":60000000000,"Timeout":10000000000}"]


2025-03-07T17:09:56.964832355+08:00 [2025/03/07 09:09:56.964 +00:00] [WARN] [client/client.go:104] ["RootCoordClient mess key not exist"] [key=rootcoord]


2025-03-07T17:09:56.964835121+08:00 [2025/03/07 09:09:56.964 +00:00] [WARN] [grpcclient/client.go:262] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]


2025-03-07T17:09:56.964839954+08:00 [2025/03/07 09:09:56.964 +00:00] [WARN] [grpcclient/client.go:473] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]


2025-03-07T17:09:56.964964256+08:00 [2025/03/07 09:09:56.964 +00:00] [WARN] [grpcclient/client.go:494] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:493\n  | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:128\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:486\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:589\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:135\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/v2/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:485\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:411\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:60\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:129\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]


2025-03-07T17:09:56.965158175+08:00 [2025/03/07 09:09:56.965 +00:00] [WARN] [client/client.go:104] ["RootCoordClient mess key not exist"] [key=rootcoord]


2025-03-07T17:09:56.965160056+08:00 [2025/03/07 09:09:56.965 +00:00] [WARN] [grpcclient/client.go:262] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]


2025-03-07T17:09:56.965165399+08:00 [2025/03/07 09:09:56.965 +00:00] [WARN] [grpcclient/client.go:480] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]


2025-03-07T17:09:56.965242031+08:00 [2025/03/07 09:09:56.965 +00:00] [WARN] [retry/retry.go:130] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:493\n  | github.com/milvus-io/milvus/pkg/v2/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:128\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:486\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:573\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:589\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:135\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/v2/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n  | \t/workspace/source/internal/distributed/proxy/service.go:485\n  | github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n  | \t/workspace/source/internal/distributed/proxy/service.go:411\n  | github.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n  | \t/workspace/source/cmd/components/proxy.go:60\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:129\n  | runtime.goexit\n  | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]


2025-03-07T17:09:57.060658134+08:00 [2025/03/07 09:09:57.060 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]


2025-03-07T17:09:57.060687797+08:00 [2025/03/07 09:09:57.060 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]


2025-03-07T17:09:57.060967179+08:00 [2025/03/07 09:09:57.060 +00:00] [INFO] [querynodev2/server.go:304] ["register metrics actions finished"]


2025-03-07T17:09:57.060972958+08:00 [2025/03/07 09:09:57.060 +00:00] [INFO] [querynodev2/server.go:314] ["QueryNode session info"] [metaPath=by-dev/meta]


2025-03-07T17:09:57.062820549+08:00 [2025/03/07 09:09:57.062 +00:00] [INFO] [datanode/service.go:266] ["initializing RootCoord client for DataNode"]


2025-03-07T17:09:57.062826516+08:00 [2025/03/07 09:09:57.062 +00:00] [INFO] [etcd/etcd_util.go:52] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[http://******:2379]"] [minVersion=1.3]


2025-03-07T17:09:57.063363628+08:00 [2025/03/07 09:09:57.063 +00:00] [INFO] [indexnode/indexnode.go:231] ["IndexNode init"] [state=Initializing]


2025-03-07T17:09:57.064412355+08:00 [2025/03/07 09:09:57.064 +00:00] [INFO] [querycoordv2/server.go:252] ["QueryCoord start init"] [meta-root-path=by-dev/meta] [address=*****:19531]


2025-03-07T17:09:57.064429332+08:00 [2025/03/07 09:09:57.064 +00:00] [INFO] [querycoordv2/server.go:247] ["register metrics actions finished"]


2025-03-07T17:09:57.065601657+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [sessionutil/session_util.go:305] ["start server"] [name=indexcoord] [address=****:13333] [id=21] [server_labels={}]


2025-03-07T17:09:57.065666269+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [sessionutil/session_util.go:305] ["start server"] [name=querynode] [address=****:21123] [id=21] [server_labels={}]


2025-03-07T17:09:57.065729070+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [sessionutil/session_util.go:305] ["start server"] [name=rootcoord] [address=****:53100] [id=21] [server_labels={}]


2025-03-07T17:09:57.065793899+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [sessionutil/session_util.go:1273] ["save server info into file"] [content="querynode-21\n"] [filePath=/tmp/milvus/server_id_1]


2025-03-07T17:09:57.065795963+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [querynodev2/server.go:171] ["QueryNode init session"] [nodeID=21] ["node address"=*****:21123]


2025-03-07T17:09:57.065832971+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [resolver/resolver_with_discoverer.go:128] ["service discover task started, listening..."] [scheme=milvus-session]


2025-03-07T17:09:57.065855941+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]


2025-03-07T17:09:57.065857829+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [rootcoord/root_coord.go:165] ["update rootcoord state"] [state=Initializing]


2025-03-07T17:09:57.065859520+08:00 [2025/03/07 09:09:57.065 +00:00] [INFO] [msgstream/mq_factory.go:244] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]


2025-03-07T17:09:57.065968131+08:00 [2025/03/07 09:09:57.065 +00:00] [ERROR] [querynodev2/server.go:345] ["QueryNode init vector storage failed"] [error="Endpoint url cannot have fully qualified paths."] [stack="github.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/workspace/source/internal/querynodev2/server.go:345\nsync.(*Once).doSlow\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/sync/once.go:74\nsync.(*Once).Do\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/sync/once.go:65\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/workspace/source/internal/querynodev2/server.go:311\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:144\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:227\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:59\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:129"]


2025-03-07T17:09:57.065993317+08:00 [2025/03/07 09:09:57.065 +00:00] [ERROR] [querynode/service.go:145] ["QueryNode init error: "] [error="Endpoint url cannot have fully qualified paths."] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/workspace/source/internal/distributed/querynode/service.go:145\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/workspace/source/internal/distributed/querynode/service.go:227\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:59\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:129"]


2025-03-07T17:09:57.066000276+08:00 [2025/03/07 09:09:57.065 +00:00] [ERROR] [components/query_node.go:60] ["QueryNode starts error"] [error="Endpoint url cannot have fully qualified paths."] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/workspace/source/cmd/components/query_node.go:60\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/workspace/source/cmd/roles/roles.go:129"]


2025-03-07T17:09:57.069184984+08:00 panic: Endpoint url cannot have fully qualified paths.


Anything else?


No response

      		
    

      
  





        



            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @zyyworktable
zyyworktable




            added
      

  kind/bug

  Issues or changes related a bug

      

  needs-triage

  Indicates an issue or PR lacks a `triage/foo` label and requires one.

  labels


      Mar 7, 2025

    

  



    


    

    

      
    

    


      

          @dosubot
dosubot
bot



            added
      

  area/kubernetes


      

  severity/critical

  Critical, lead to crash, data missing, wrong result, function totally doesn't work.

  labels


      Mar 7, 2025

    

  








      

  
      



  

  @dosubot

    
      DosuBot
    




  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  


  

    

      

      
            dosubot
  bot

      

      

      commented


        Mar 7, 2025


      
    


  





      


        


  
    

      
          

@zyyworktable 你好! 我是Dosu，一个专注于解决bug、回答问题和帮助您成为贡献者的机器人。在等待人类维护者的过程中，我将尽力帮助您。



A similar issue was discussed in a thread titled "milvus-v2.4.6集群启动报错datacoord报错，求助" where the error "Endpoint url cannot have fully qualified paths" was related to MinIO settings. The issue was resolved by ensuring that the MinIO configuration did not include fully qualified paths in the endpoint URL. This is often related to how the MinIO address is set in the configuration, which should not include paths beyond the base URL [1].


To resolve this, check your MinIO configuration in the Kubernetes YAML files. Ensure that the minio.address is set correctly without any additional paths. The address should be a simple hostname or IP address, such as minio:9000, without any additional path components. This should help in resolving the error you're encountering.



To continue talking to Dosu, mention @dosu.




Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @xiaofan-luan





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Collaborator


      

  


  

    

  





      


        


  
    

      
          
QueryNode init vector storage failed"] [error="Endpoint url cannot have fully qualified paths."]


I think you need to check your minio path

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @yanliang567





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Contributor


      

  


  

    

  





      


        


  
    

      
          
@zyyworktable BTW, as you already have k8s, it is recommended to use milvus helm or milvus operator to deploy a milvus standalone/cluster. it is much easier and more maintainable.

/assign @zyyworktable

/unassign

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
            


    

    

      
    

    


      

          @yanliang567
yanliang567




            added
      

  help wanted

  Extra attention is needed

    and removed
      

  needs-triage

  Indicates an issue or PR lacks a `triage/foo` label and requires one.

      

  severity/critical

  Critical, lead to crash, data missing, wrong result, function totally doesn't work.

  labels


      Mar 9, 2025

    

  








      

  
      



  

  @Niceies





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  


  

    

      

      
            Niceies
  

      

      

      commented


        Mar 11, 2025


      
    


  





      


        


  
    

      
          
Please refer to this reply, it may also be related

#28032 (comment)

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    
















  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

      

        

  
      @zyyworktable
  
    zyyworktable
    

      








      


  


  

    Labels
  



    

      

    area/kubernetes

  

    help wanted

  Extra attention is needed
  

    kind/bug

  Issues or changes related a bug








      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    

      
        

          
    

    Development
  




            
    
No branches or pull requests







      
    

  




      

    

  
      

  

    

      4 participants
    

    

        
          @Niceies 
        
          @zyyworktable 
        
          @yanliang567 
        
          @xiaofan-luan