Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docker部署nacos集群nginx负载均衡，服务注册正常，但是grpc报错 #10707

Closed
lidaoyang opened this issue Jun 28, 2023 · 18 comments
Closed

docker部署nacos集群nginx负载均衡，服务注册正常，但是grpc报错 #10707

lidaoyang opened this issue Jun 28, 2023 · 18 comments
Labels
kind/question Category issues related to questions or problems

Comments

@lidaoyang
Copy link
Contributor

lidaoyang commented Jun 28, 2023 •
edited

nacos-server 版本：2.2.3
nacos-client：2.1.0（spring-cloud-alibaba.version 2.2.9）
nginx容器IP：
nginx配置：172.2.1.6

  upstream nacos.grpc_server {
        server 172.2.1.3:9848;
        server 172.2.1.4:9848;
        server 172.2.1.5:9848;
    }

 server {
        listen 1080 so_keepalive=on;
        proxy_responses 1;
        proxy_timeout 20s;
        proxy_pass nacos.grpc_server;

    }
}
   upstream nacos.jrmall.cn {
        server 172.2.1.3:8848;
        server 172.2.1.4:8848;
        server 172.2.1.5:8848;
   }

错误日志：

2023-06-27 07:27:22.412 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [728e3d5f-4ad0-4f17-9a54-352a82c1f782] Server healthy check fail, currentConnection = 1687847317491_172.2.1.6_52294
2023-06-27 07:27:22.413 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [728e3d5f-4ad0-4f17-9a54-352a82c1f782] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-27 07:27:22.703 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [728e3d5f-4ad0-4f17-9a54-352a82c1f782] Success to connect a server [juru.nacos.jrmall.cn:80], connectionId = 1687850843010_172.2.1.6_46518
2023-06-27 07:27:22.703 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [728e3d5f-4ad0-4f17-9a54-352a82c1f782] Abandon prev connection, server is juru.nacos.jrmall.cn:80, connectionId is 1687847317491_172.2.1.6_52294
2023-06-27 07:27:22.703 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] Close current connection 1687847317491_172.2.1.6_52294
2023-06-27 07:27:22.708 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [728e3d5f-4ad0-4f17-9a54-352a82c1f782] Notify disconnected event to listeners
2023-06-27 07:27:22.709 [system-service] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] Grpc connection disconnect, mark to redo
2023-06-27 07:27:22.711 [system-service] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] mark to redo completed
2023-06-27 07:27:22.712 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [728e3d5f-4ad0-4f17-9a54-352a82c1f782] Notify connected event to listeners.
2023-06-27 07:27:22.712 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.naming - [] Grpc connection connect
2023-06-27 07:27:22.713 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-1106] WARN  c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687847317491_172.2.1.6_52294]Ignore error event,isRunning:true,isAbandon=true
2023-06-27 07:27:25.120 [system-service] [com.alibaba.nacos.client.naming.grpc.redo.0] INFO  com.alibaba.nacos.client.naming - [] Redo subscriber operation REGISTER for DEFAULT_GROUP@@providers:com.jrmall.cate.service.CategoryServiceI#
2023-06-27 07:27:27.756 [system-service] [com.alibaba.nacos.client.naming.grpc.redo.0] INFO  com.alibaba.nacos.client.naming - [] Redo subscriber operation REGISTER for DEFAULT_GROUP@@providers:com.jrmall.cate.service.CategoryServiceI::#
2023-06-27 07:27:28.349 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-1111] INFO  com.alibaba.nacos.common.remote.client - [] [728e3d5f-4ad0-4f17-9a54-352a82c1f782] Receive server push request, request = NotifySubscriberRequest, requestId = 101
2023-06-27 07:27:28.349 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-1111] INFO  com.alibaba.nacos.common.remote.client - [] [728e3d5f-4ad0-4f17-9a54-352a82c1f782] Ack server push request, request = NotifySubscriberRequest, requestId = 101
2023-06-27 07:27:28.351 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-1112] INFO  com.alibaba.nacos.common.remote.client - [] [728e3d5f-4ad0-4f17-9a54-352a82c1f782] Receive server push request, request = NotifySubscriberRequest, requestId = 102
2023-06-27 07:27:28.351 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-1112] INFO  com.alibaba.nacos.common.remote.client - [] [728e3d5f-4ad0-4f17-9a54-352a82c1f782] Ack server push request, request = NotifySubscriberRequest, requestId = 102
2023-06-27 07:32:52.829 [system-service] [com.alibaba.nacos.client.naming.updater.0] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ServiceQueryRequest{headers={}, requestId='null'}, retryTimes = 0, errorMessage = java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 73991 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@289e35a9[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@70019abe, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@7d6b2b85, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@2be0e8a}}}}}]]
2023-06-27 07:32:52.836 [system-service] [com.alibaba.nacos.client.naming.updater.0] WARN  com.alibaba.nacos.client.naming - [] [NA] failed to update serviceName: DEFAULT_GROUP@@category-service
com.alibaba.nacos.api.exception.NacosException: Request nacos server failed: 
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:288)
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.queryInstancesOfService(NamingGrpcClientProxy.java:168)
        at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.queryInstancesOfService(NamingClientProxyDelegate.java:110)
        at com.alibaba.nacos.client.naming.core.ServiceInfoUpdateService$UpdateTask.run(ServiceInfoUpdateService.java:185)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.nacos.api.exception.NacosException: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 73991 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@289e35a9[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@70019abe, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@7d6b2b85, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@2be0e8a}}}}}]]
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:78)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:653)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:631)
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:278)
        ... 10 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 73991 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@289e35a9[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@70019abe, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@7d6b2b85, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@2be0e8a}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:76)
        ... 13 common frames omitted
2023-06-27 07:32:52.913 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [4f9c66cb-d8cb-463a-a1ec-6233d9b2119d] Server check success, currentServer is juru.nacos.jrmall.cn:80 
2023-06-27 15:43:41.841 [system-service] [com.alibaba.nacos.client.Worker] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ConfigBatchListenRequest{headers={}, requestId='null'}, retryTimes = 0, errorMessage = java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 72668 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@656f31ae[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@70019abe, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@7d6b2b85, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@2be0e8a}}}}}]]
2023-06-27 15:43:41.842 [system-service] [com.alibaba.nacos.client.Worker] ERROR com.alibaba.nacos.client.config.impl.ClientWorker - [] Async listen config change error 
com.alibaba.nacos.api.exception.NacosException: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 72668 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@656f31ae[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@70019abe, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@7d6b2b85, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@2be0e8a}}}}}]]
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:78)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:653)
        at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.requestProxy(ClientWorker.java:1005)
        at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.requestProxy(ClientWorker.java:986)
        at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.executeConfigListen(ClientWorker.java:777)
        at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.lambda$startInternal$2(ClientWorker.java:695)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 72668 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@656f31ae[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@70019abe, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@7d6b2b85, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@2be0e8a}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:76)
        ... 12 common frames omitted
2023-06-27 15:43:41.963 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [2a301e02-e3f7-4d8e-844b-d222ab2050fc_config-0] Server check success, currentServer is juru.nacos.jrmall.cn:80 
2023-06-27 15:45:04.896 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [4f9c66cb-d8cb-463a-a1ec-6233d9b2119d] Server healthy check fail, currentConnection = 1687847314931_172.2.1.6_38872
2023-06-27 15:45:04.896 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [4f9c66cb-d8cb-463a-a1ec-6233d9b2119d] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-27 15:45:05.647 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [4f9c66cb-d8cb-463a-a1ec-6233d9b2119d] Success to connect a server [juru.nacos.jrmall.cn:80], connectionId = 1687880705965_172.2.1.6_48830
2023-06-27 15:45:05.647 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [4f9c66cb-d8cb-463a-a1ec-6233d9b2119d] Abandon prev connection, server is juru.nacos.jrmall.cn:80, connectionId is 1687847314931_172.2.1.6_38872
2023-06-27 15:45:05.647 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] Close current connection 1687847314931_172.2.1.6_38872
2023-06-27 15:45:05.647 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [4f9c66cb-d8cb-463a-a1ec-6233d9b2119d] Notify disconnected event to listeners
2023-06-27 15:45:05.648 [system-service] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] Grpc connection disconnect, mark to redo
2023-06-27 15:45:05.648 [system-service] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] mark to redo completed
2023-06-27 15:45:05.648 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [4f9c66cb-d8cb-463a-a1ec-6233d9b2119d] Notify connected event to listeners.
2023-06-27 15:45:05.648 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.naming - [] Grpc connection connect
@985492783
Copy link
Contributor

grpc端口开放了吗

@lidaoyang
Copy link
Contributor Author

grpc端口开放了吗

开了，我服务器上貌似好了，现在本地注册到服务器的机器IP，可以正常注册，但是会报grpc错误，等会发一下日志

@lidaoyang
Copy link
Contributor Author

grpc端口开放了吗

下面是一个服务报的错误，访问是没有问题的


2023-06-28 14:09:34.412 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [d0bbc26e-4342-4616-b19f-e4c7ecb22a9f_config-0] Connected,notify listen context...
2023-06-28 14:09:34.415 [oauth2] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-118] WARN  c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687932026271_172.2.1.6_48288]Ignore error event,isRunning:true,isAbandon=true
2023-06-28 14:09:48.197 [oauth2] [http-nio-9090-exec-9] INFO  com.jrmall.oauth2.filter.LogMDCFilter - [d14e7c33054d4afe9714317568888f6a] 请求开始,traceId:d14e7c33054d4afe9714317568888f6a,url:/oauth/token
2023-06-28 14:09:48.216 [oauth2] [http-nio-9090-exec-9] DEBUG com.jrmall.oauth2.config.SmsCodeTokenGranter - [d14e7c33054d4afe9714317568888f6a] Getting access token for: jrmall-admin-client
2023-06-28 14:09:54.481 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [201e9b23-8ad7-4cc9-aac2-9d28214d4aa5] Server healthy check fail, currentConnection = 1687932058429_172.2.1.6_49374
2023-06-28 14:09:54.483 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [201e9b23-8ad7-4cc9-aac2-9d28214d4aa5] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-28 14:10:00.566 [oauth2] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 3078 milliseconds, 38012 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@5e1326b7[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@24d61e4, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@2149594a, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f1e58ca}}}}}]]
	at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
	at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:192)
	at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:302)
	at com.alibaba.nacos.common.remote.client.RpcClient.reconnect(RpcClient.java:521)
	at com.alibaba.nacos.common.remote.client.RpcClient.lambda$start$2(RpcClient.java:369)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2023-06-28 14:10:06.357 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [8849a19d-d84f-4697-9d32-83aa176328f8] Server healthy check fail, currentConnection = 1687932051405_172.2.1.6_50216
2023-06-28 14:10:06.357 [oauth2] [com.alibaba.nacos.client.naming.updater.0] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ServiceQueryRequest{headers={}, requestId='null'}, retryTimes = 0, errorMessage = java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 2789 milliseconds, 607068 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@4ea4568c[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@24d61e4, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@2149594a, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f1e58ca}}}}}]]
2023-06-28 14:10:13.076 [oauth2] [com.alibaba.nacos.client.naming.updater.0] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ServiceQueryRequest{headers={accessToken=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJuYWNvcyIsImV4cCI6MTY4Nzk1MDA1N30.FxlttjPpKgYC8CSFux6YgMvETprBJ5bhSEzew6D9OVo, app=oauth2}, requestId='null'}, retryTimes = 0, errorMessage = Client not connected, current status:UNHEALTHY
2023-06-28 14:10:13.078 [oauth2] [com.alibaba.nacos.client.naming.updater.0] WARN  com.alibaba.nacos.client.naming - [] [NA] failed to update serviceName: DEFAULT_GROUP@@system-service
com.alibaba.nacos.api.exception.NacosException: Request nacos server failed: 
	at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:288)
	at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.queryInstancesOfService(NamingGrpcClientProxy.java:168)
	at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.queryInstancesOfService(NamingClientProxyDelegate.java:110)
	at com.alibaba.nacos.client.naming.core.ServiceInfoUpdateService$UpdateTask.run(ServiceInfoUpdateService.java:185)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: com.alibaba.nacos.api.exception.NacosException: Client not connected, current status:UNHEALTHY
	at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:651)
	at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:631)
	at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:278)
	... 11 common frames omitted
2023-06-28 14:10:13.078 [oauth2] [com.alibaba.nacos.client.naming.updater.0] WARN  com.alibaba.nacos.client.naming - [] [NA] failed to update serviceName: DEFAULT_GROUP@@oauth2
com.alibaba.nacos.api.exception.NacosException: Request nacos server failed: 
	at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:288)
	at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.queryInstancesOfService(NamingGrpcClientProxy.java:168)
	at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.queryInstancesOfService(NamingClientProxyDelegate.java:110)
	at com.alibaba.nacos.client.naming.core.ServiceInfoUpdateService$UpdateTask.run(ServiceInfoUpdateService.java:185)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: com.alibaba.nacos.api.exception.NacosException: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 2789 milliseconds, 607068 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@4ea4568c[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@24d61e4, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@2149594a, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f1e58ca}}}}}]]
	at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:78)
	at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:653)
	at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:631)
	at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:278)
	... 11 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 2789 milliseconds, 607068 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@4ea4568c[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@24d61e4, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@2149594a, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f1e58ca}}}}}]]
	at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
	at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:76)
	... 14 common frames omitted
2023-06-28 14:10:13.076 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [8849a19d-d84f-4697-9d32-83aa176328f8] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-28 14:10:15.105 [oauth2] [http-nio-9090-exec-9] INFO  com.jrmall.oauth2.filter.LogMDCFilter - [d14e7c33054d4afe9714317568888f6a] 请求结束,traceId:d14e7c33054d4afe9714317568888f6a
2023-06-28 14:10:15.106 [oauth2] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-123] WARN  c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687932058429_172.2.1.6_49374]Ignore error event,isRunning:false,isAbandon=false
2023-06-28 14:10:15.205 [oauth2] [com.alibaba.nacos.client.naming.updater.0] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ServiceQueryRequest{headers={accessToken=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJuYWNvcyIsImV4cCI6MTY4Nzk1MDA1MX0.9GusX3_5zFgpReQasScf0U2-HPKMyvsfdzC9vDNeXy4, app=oauth2}, requestId='null'}, retryTimes = 0, errorMessage = Client not connected, current status:UNHEALTHY
2023-06-28 14:10:15.205 [oauth2] [com.alibaba.nacos.client.naming.updater.0] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ServiceQueryRequest{headers={accessToken=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJuYWNvcyIsImV4cCI6MTY4Nzk1MDA1N30.FxlttjPpKgYC8CSFux6YgMvETprBJ5bhSEzew6D9OVo, app=oauth2}, requestId='null'}, retryTimes = 0, errorMessage = Client not connected, current status:UNHEALTHY
2023-06-28 14:10:15.210 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [201e9b23-8ad7-4cc9-aac2-9d28214d4aa5] Success to connect a server [juru.nacos.jrmall.cn:80], connectionId = 1687932607050_172.2.1.6_54626
2023-06-28 14:10:15.211 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [201e9b23-8ad7-4cc9-aac2-9d28214d4aa5] Abandon prev connection, server is juru.nacos.jrmall.cn:80, connectionId is 1687932058429_172.2.1.6_49374
2023-06-28 14:10:15.211 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] Close current connection 1687932058429_172.2.1.6_49374
2023-06-28 14:10:15.211 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [201e9b23-8ad7-4cc9-aac2-9d28214d4aa5] Notify disconnected event to listeners
2023-06-28 14:10:15.211 [oauth2] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] Grpc connection disconnect, mark to redo
2023-06-28 14:10:15.213 [oauth2] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] mark to redo completed
2023-06-28 14:10:15.213 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [201e9b23-8ad7-4cc9-aac2-9d28214d4aa5] Notify connected event to listeners.
2023-06-28 14:10:15.213 [oauth2] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.naming - [] Grpc connection connect

@lidaoyang
Copy link
Contributor Author

本地服务注册到局域网中的docker nacos中，grpc连接的是docker容器的IP，dubbo注册的是我的本机IP，
172.2.1.6 是nginx docker容器的IP

2023-06-28 16:26:57.280 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [79be3451-2ab7-463d-9951-1bddbcfcb58b] Server healthy check fail, currentConnection = 1687935787927_172.2.1.6_48692
2023-06-28 16:26:57.293 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-962] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687935736550_**172.2.1.6**_34514]Request stream error, switch server,error={}
com.alibaba.nacos.shaded.io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at com.alibaba.nacos.shaded.io.grpc.Status.asRuntimeException(Status.java:533)
	at com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:449)
	at com.alibaba.nacos.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.alibaba.nacos.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at com.alibaba.nacos.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.alibaba.nacos.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:521)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:641)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:529)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:703)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:692)
	at com.alibaba.nacos.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at com.alibaba.nacos.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2023-06-28 16:26:57.250 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-1085] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687935787927_172.2.1.6_48692]Request stream error, switch server,error={}
com.alibaba.nacos.shaded.io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at com.alibaba.nacos.shaded.io.grpc.Status.asRuntimeException(Status.java:533)
	at com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:449)
	at com.alibaba.nacos.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.alibaba.nacos.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at com.alibaba.nacos.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.alibaba.nacos.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:521)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:641)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:529)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:703)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:692)
	at com.alibaba.nacos.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at com.alibaba.nacos.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2023-06-28 16:26:57.303 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [ba778379-e731-4a2b-860f-2a2fcd21ec4b_config-0] Server healthy check fail, currentConnection = 1687935736550_172.2.1.6_34514
2023-06-28 16:26:57.316 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-968] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687935751203_172.2.1.6_56634]Request stream error, switch server,error={}
com.alibaba.nacos.shaded.io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at com.alibaba.nacos.shaded.io.grpc.Status.asRuntimeException(Status.java:533)
	at com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:449)
	at com.alibaba.nacos.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.alibaba.nacos.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at com.alibaba.nacos.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.alibaba.nacos.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:521)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:641)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:529)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:703)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:692)
	at com.alibaba.nacos.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at com.alibaba.nacos.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2023-06-28 16:26:57.335 [system-service] [com.alibaba.nacos.client.Worker] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ConfigBatchListenRequest{headers={}, requestId='null'}, retryTimes = 0, errorMessage = java.util.concurrent.ExecutionException: com.alibaba.nacos.shaded.io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
2023-06-28 16:26:57.336 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [79be3451-2ab7-463d-9951-1bddbcfcb58b] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-28 16:26:57.337 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [ba778379-e731-4a2b-860f-2a2fcd21ec4b_config-0] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-28 16:26:57.439 [system-service] [com.alibaba.nacos.client.Worker] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ConfigBatchListenRequest{headers={}, requestId='null'}, retryTimes = 1, errorMessage = Client not connected, current status:UNHEALTHY
2023-06-28 16:26:57.543 [system-service] [com.alibaba.nacos.client.Worker] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ConfigBatchListenRequest{headers={}, requestId='null'}, retryTimes = 2, errorMessage = Client not connected, current status:UNHEALTHY
2023-06-28 16:26:57.544 [system-service] [com.alibaba.nacos.client.Worker] ERROR com.alibaba.nacos.client.config.impl.ClientWorker - [] Async listen config change error 
com.alibaba.nacos.api.exception.NacosException: Client not connected, current status:UNHEALTHY
	at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:651)
	at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.requestProxy(ClientWorker.java:1005)
	at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.requestProxy(ClientWorker.java:986)
	at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.executeConfigListen(ClientWorker.java:777)
	at com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient.lambda$startInternal$2(ClientWorker.java:695)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2023-06-28 16:26:57.963 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [ba778379-e731-4a2b-860f-2a2fcd21ec4b_config-0] Success to connect a server [juru.nacos.jrmall.cn:80], connectionId = 1687940811636_172.2.1.6_52100
2023-06-28 16:26:57.963 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [ba778379-e731-4a2b-860f-2a2fcd21ec4b_config-0] Abandon prev connection, server is juru.nacos.jrmall.cn:80, connectionId is 1687935736550_172.2.1.6_34514
2023-06-28 16:26:57.963 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] Close current connection 1687935736550_172.2.1.6_34514
2023-06-28 16:26:57.967 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [ba778379-e731-4a2b-860f-2a2fcd21ec4b_config-0] Notify disconnected event to listeners
2023-06-28 16:26:57.968 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [ba778379-e731-4a2b-860f-2a2fcd21ec4b_config-0] DisConnected,clear listen context...
2023-06-28 16:26:57.969 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [ba778379-e731-4a2b-860f-2a2fcd21ec4b_config-0] Notify connected event to listeners.
2023-06-28 16:26:57.969 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [ba778379-e731-4a2b-860f-2a2fcd21ec4b_config-0] Connected,notify listen context...
2023-06-28 16:26:58.569 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-938] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687935786932_172.2.1.6_39704]Request stream error, switch server,error={}
com.alibaba.nacos.shaded.io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at com.alibaba.nacos.shaded.io.grpc.Status.asRuntimeException(Status.java:533)
	at com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:449)
	at com.alibaba.nacos.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.alibaba.nacos.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at com.alibaba.nacos.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.alibaba.nacos.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:521)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:641)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:529)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:703)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:692)
	at com.alibaba.nacos.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at com.alibaba.nacos.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2023-06-28 16:26:58.571 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [e4d7b858-6be8-41f0-9307-6829d2be980e] Server healthy check fail, currentConnection = 1687935786932_172.2.1.6_39704
2023-06-28 16:26:58.571 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [e4d7b858-6be8-41f0-9307-6829d2be980e] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-28 16:26:58.715 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [79be3451-2ab7-463d-9951-1bddbcfcb58b] Success to connect a server [juru.nacos.jrmall.cn:80], connectionId = 1687940811470_172.2.1.6_38268
2023-06-28 16:26:58.716 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [79be3451-2ab7-463d-9951-1bddbcfcb58b] Abandon prev connection, server is juru.nacos.jrmall.cn:80, connectionId is 1687935787927_172.2.1.6_48692
2023-06-28 16:26:58.716 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] Close current connection 1687935787927_172.2.1.6_48692
2023-06-28 16:26:58.717 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [79be3451-2ab7-463d-9951-1bddbcfcb58b] Notify disconnected event to listeners
2023-06-28 16:26:58.717 [system-service] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] Grpc connection disconnect, mark to redo
2023-06-28 16:26:58.720 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-961] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687935751808_172.2.1.6_55064]Request stream error, switch server,error={}
com.alibaba.nacos.shaded.io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at com.alibaba.nacos.shaded.io.grpc.Status.asRuntimeException(Status.java:533)
	at com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:449)
	at com.alibaba.nacos.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.alibaba.nacos.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at com.alibaba.nacos.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.alibaba.nacos.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.alibaba.nacos.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:521)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:641)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:529)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:703)
	at com.alibaba.nacos.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:692)
	at com.alibaba.nacos.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at com.alibaba.nacos.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

@985492783
Copy link
Contributor

985492783 commented Jun 28, 2023 •
edited

可以看一下这个的配置 #10452

@985492783
Copy link
Contributor

以及#9004 nginx做nacos负载均衡确实缺一个最佳实践

@lidaoyang
Copy link
Contributor Author

可以看一下这个的配置 #10452

是的，我看了上面的issue，我服务器上现在没有报错了，都是用docker他们网络是通的。我本地连接的是服务器主机IP和NGINX docker映射端口，服务注册上去是没有问题，但是过一段时间就报grpc错误，看到上面的IP地址是nginx 容器的IP，本地不知道怎么连服务器了，本地和服务器都是一个局域网，就是一个是容器，一个是直接运行的

@lidaoyang
Copy link
Contributor Author

今天发现服务器docker容器里的日志又报错了，日志里可以看到 Connection refused: juru.nacos.jrmall.cn/172.2.1.6:1080 拒绝访问的，80端口是正常的，很不解，哪位大神指点一下

.2.2.jar!/, jar:file:/home/admin/app/category-service/target/category-service.jar!/BOOT-INF/lib/mybatis-spring-boot-autoconfigure-2.2.2.jar!/, jar:file:/home/admin/app/category-service/target/category-service.jar!/BOOT-INF/lib/pagehelper-spring-boot-autoconfigure-1.4.6.jar!/, jar:file:/home/admin/app/category-service/target/category-service.jar!/BOOT-INF/lib/pagehelper-5.3.2.jar!/, jar:file:/home/admin/app/category-service/target/category-service.jar!/BOOT-INF/lib/slf4j-api-1.7.36.jar!/, jar:file:/home/admin/app/category-service/target/category-service.jar!/BOOT-INF/lib/logback-core-1.2.11.jar!/, jar:file:/home/admin/app/category-service/target/category-service.jar!/BOOT-INF/lib/logback-classic-1.2.11.jar!/, jar:file:/home/admin/app/category-service/target/category-service.jar!/BOOT-INF/lib/lombok-1.18.24.jar!/]

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::       (v2.3.12.RELEASE)

2023-06-27 06:30:30.427 [category-service] [main] DEBUG o.springframework.boot.env.OriginTrackedYamlLoader - [] Loading from YAML: Byte array resource [mysql-db-config.yml]
2023-06-27 06:30:30.431 [category-service] [main] DEBUG o.springframework.boot.env.OriginTrackedYamlLoader - [] Merging document (no matchers set): {spring={datasource={druid={url=jdbc:mysql://172.2.1.2:3306/shop_new?useUnicode=true&useSSL=false&characterEncoding=utf8&serverTimezone=UTC&allowPublicKeyRetrieval=true, driver-class-name=com.mysql.cj.jdbc.Driver, username=shop_new, password=T5cpZEmwhf2SGNn3, connection-properties=druid.stat.mergeSql=true;druid.stat.slowSqlMillis=500, filters=stat,wall,log4j2, initial-size=5, min-idle=5, max-active=100, max-wait=60000, max-pool-prepared-statement-per-connection-size=0, min-evictable-idle-time-millis=300000, pool-prepared-statements=false, test-on-borrow=false, test-on-return=false, test-while-idle=true, time-between-eviction-runs-millis=60000, use-global-data-source-stat=true, validation-query=SELECT 1, web-stat-filter={enabled=true, url-pattern=/*}, filter={stat={enabled=true, log-slow-sql=true, slow-sql-millis=10000, merge-sql=true}}}}}}
2023-06-27 06:30:30.432 [category-service] [main] DEBUG o.springframework.boot.env.OriginTrackedYamlLoader - [] Loaded 1 document from YAML resource: Byte array resource [mysql-db-config.yml]
2023-06-27 06:30:30.445 [category-service] [main] DEBUG o.springframework.boot.env.OriginTrackedYamlLoader - [] Loading from YAML: Byte array resource [redis-config.yml]
2023-06-27 06:30:30.448 [category-service] [main] DEBUG o.springframework.boot.env.OriginTrackedYamlLoader - [] Merging document (no matchers set): {spring={redis={password=xhndmm86huyn, host=172.16.85.105, port=6379, timeout=30000, database=0, jedis={pool={max-active=200, max-wait=-1, max-idle=10, min-idle=0, time-between-eviction-runs=-1}}}}}
2023-06-27 06:30:30.448 [category-service] [main] DEBUG o.springframework.boot.env.OriginTrackedYamlLoader - [] Loaded 1 document from YAML resource: Byte array resource [redis-config.yml]
2023-06-27 06:30:30.456 [category-service] [main] DEBUG o.springframework.boot.env.OriginTrackedYamlLoader - [] Loading from YAML: Byte array resource [oauth2-config.yml]
2023-06-27 06:30:30.459 [category-service] [main] DEBUG o.springframework.boot.env.OriginTrackedYamlLoader - [] Merging document (no matchers set): {security={oauth2={client={access-token-uri=http://juru.new.jrmall.cn/oauth2/oauth/token, client-authentication-scheme=form, client-id=jrmall-admin-client, client-secret=Ju8FwE60OwSiKDoRi4sClYUi97kLDj, grant-type=password, grant-type-sms=sms_code, grant-type-refresh=refresh_token}, resource={token-info-uri=http://juru.new.jrmall.cn/oauth2/oauth/check_token}}}}
2023-06-27 06:30:30.459 [category-service] [main] DEBUG o.springframework.boot.env.OriginTrackedYamlLoader - [] Loaded 1 document from YAML resource: Byte array resource [oauth2-config.yml]
2023-06-27 06:30:30.467 [category-service] [main] WARN  c.a.cloud.nacos.client.NacosPropertySourceBuilder - [] Ignore the empty nacos configuration and get it based on dataId[category-service] & group[DEFAULT_GROUP]
2023-06-27 06:30:30.474 [category-service] [main] WARN  c.a.cloud.nacos.client.NacosPropertySourceBuilder - [] Ignore the empty nacos configuration and get it based on dataId[category-service.properties] & group[DEFAULT_GROUP]
2023-06-27 06:30:30.482 [category-service] [main] WARN  c.a.cloud.nacos.client.NacosPropertySourceBuilder - [] Ignore the empty nacos configuration and get it based on dataId[category-service-prod.properties] & group[DEFAULT_GROUP]
2023-06-27 06:30:30.524 [category-service] [main] INFO  com.jrmall.cate.CategoryServiceApplication - [] The following profiles are active: prod
2023-06-27 06:30:33.233 [category-service] [main] INFO  org.apache.coyote.http11.Http11NioProtocol - [] Initializing ProtocolHandler ["http-nio-8080"]
2023-06-27 06:30:33.234 [category-service] [main] INFO  org.apache.catalina.core.StandardService - [] Starting service [Tomcat]
2023-06-27 06:30:33.234 [category-service] [main] INFO  org.apache.catalina.core.StandardEngine - [] Starting Servlet engine: [Apache Tomcat/9.0.46]
2023-06-27 06:30:33.293 [category-service] [main] INFO  o.a.c.core.ContainerBase.[Tomcat].[localhost].[/] - [] Initializing Spring embedded WebApplicationContext
2023-06-27 06:30:33.823 [category-service] [main] INFO  c.a.n.p.auth.spi.client.ClientAuthPluginManager - [] [ClientAuthPluginManager] Load ClientAuthService com.alibaba.nacos.client.auth.impl.NacosClientAuthServiceImpl success.
2023-06-27 06:30:33.823 [category-service] [main] INFO  c.a.n.p.auth.spi.client.ClientAuthPluginManager - [] [ClientAuthPluginManager] Load ClientAuthService com.alibaba.nacos.client.auth.ram.RamClientAuthServiceImpl success.
2023-06-27 06:30:33.937 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [RpcClientFactory] create a new rpc client of 1c3880d3-b203-4658-8a44-d9b14971943a_config-0
2023-06-27 06:30:33.937 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] RpcClient init label, labels = {module=config, Vipserver-Tag=null, source=sdk, Amory-Tag=null, Location-Tag=null, taskId=0, AppName=category-service}
2023-06-27 06:30:33.937 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Register server push request handler:com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$$Lambda$284/1904324159
2023-06-27 06:30:33.937 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Register server push request handler:com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$$Lambda$285/1176735295
2023-06-27 06:30:33.937 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Registry connection listener to current client:com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$1
2023-06-27 06:30:33.937 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] RpcClient init, ServerListFactory = com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$2
2023-06-27 06:30:33.938 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Try to connect to server on start up, server: {serverIp = 'juru.nacos.jrmall.cn', server main port = 80}
2023-06-27 06:30:34.053 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Success to connect to server [juru.nacos.jrmall.cn:80] on start up, connectionId = 1687847433533_172.2.1.6_56312
2023-06-27 06:30:34.053 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Register server push request handler:com.alibaba.nacos.common.remote.client.RpcClient$ConnectResetRequestHandler
2023-06-27 06:30:34.053 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Notify connected event to listeners.
2023-06-27 06:30:34.053 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Register server push request handler:com.alibaba.nacos.common.remote.client.RpcClient$$Lambda$294/2008966511
2023-06-27 06:30:34.053 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Connected,notify listen context...
2023-06-27 06:30:34.592 [category-service] [main] INFO  c.a.n.p.auth.spi.client.ClientAuthPluginManager - [] [ClientAuthPluginManager] Load ClientAuthService com.alibaba.nacos.client.auth.impl.NacosClientAuthServiceImpl success.
2023-06-27 06:30:34.592 [category-service] [main] INFO  c.a.n.p.auth.spi.client.ClientAuthPluginManager - [] [ClientAuthPluginManager] Load ClientAuthService com.alibaba.nacos.client.auth.ram.RamClientAuthServiceImpl success.
2023-06-27 06:30:34.705 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [RpcClientFactory] create a new rpc client of 3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0
2023-06-27 06:30:34.706 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] RpcClient init label, labels = {module=config, Vipserver-Tag=null, source=sdk, Amory-Tag=null, Location-Tag=null, taskId=0, AppName=category-service}
2023-06-27 06:30:34.706 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Register server push request handler:com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$$Lambda$284/1904324159
2023-06-27 06:30:34.706 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Register server push request handler:com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$$Lambda$285/1176735295
2023-06-27 06:30:34.706 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Registry connection listener to current client:com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$1
2023-06-27 06:30:34.706 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] RpcClient init, ServerListFactory = com.alibaba.nacos.client.config.impl.ClientWorker$ConfigRpcTransportClient$2
2023-06-27 06:30:34.707 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Try to connect to server on start up, server: {serverIp = 'juru.nacos.jrmall.cn', server main port = 80}
2023-06-27 06:30:35.149 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Success to connect to server [juru.nacos.jrmall.cn:80] on start up, connectionId = 1687847434811_172.2.1.6_37568
2023-06-27 06:30:35.149 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Register server push request handler:com.alibaba.nacos.common.remote.client.RpcClient$ConnectResetRequestHandler
2023-06-27 06:30:35.149 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Notify connected event to listeners.
2023-06-27 06:30:35.150 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Register server push request handler:com.alibaba.nacos.common.remote.client.RpcClient$$Lambda$294/2008966511
2023-06-27 06:30:35.150 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Connected,notify listen context...
2023-06-27 06:30:35.255 [category-service] [main] INFO  c.a.d.s.b.a.DruidDataSourceAutoConfigure - [] Init DruidDataSource
2023-06-27 06:30:36.059 [category-service] [main] INFO  com.alibaba.druid.pool.DruidDataSource - [] {dataSource-1} inited
 _ _   |_  _ _|_. ___ _ |    _ 
| | |\/|_)(_| | |_\  |_)||_|_\ 
     /               |         
                        3.5.3 
2023-06-27 06:30:37.992 [category-service] [main] INFO  org.apache.coyote.http11.Http11NioProtocol - [] Starting ProtocolHandler ["http-nio-8080"]
2023-06-27 06:30:38.356 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.CacheData - [] nacos.cache.data.init.snapshot = true 
2023-06-27 06:30:38.357 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [fixed-dubbo-prod-juru.nacos.jrmall.cn_80] [subscribe] category-service.configurators+dubbo+dubbo-prod
2023-06-27 06:30:38.368 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.CacheData - [] [fixed-dubbo-prod-juru.nacos.jrmall.cn_80] [add-listener] ok, tenant=dubbo-prod, dataId=category-service.configurators, group=dubbo, cnt=1
2023-06-27 06:30:38.376 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [fixed-dubbo-prod-juru.nacos.jrmall.cn_80] [subscribe] com.jrmall.cate.service.CategoryServiceI::.configurators+dubbo+dubbo-prod
2023-06-27 06:30:38.376 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.CacheData - [] [fixed-dubbo-prod-juru.nacos.jrmall.cn_80] [add-listener] ok, tenant=dubbo-prod, dataId=com.jrmall.cate.service.CategoryServiceI::.configurators, group=dubbo, cnt=1
2023-06-27 06:30:38.584 [category-service] [main] INFO  com.alibaba.nacos.client.naming - [] initializer namespace from System Property : null
2023-06-27 06:30:38.585 [category-service] [main] INFO  com.alibaba.nacos.client.naming - [] initializer namespace from System Environment :null
2023-06-27 06:30:38.585 [category-service] [main] INFO  com.alibaba.nacos.client.naming - [] initializer namespace from System Property :null
2023-06-27 06:30:38.597 [category-service] [main] INFO  c.a.n.p.auth.spi.client.ClientAuthPluginManager - [] [ClientAuthPluginManager] Load ClientAuthService com.alibaba.nacos.client.auth.impl.NacosClientAuthServiceImpl success.
2023-06-27 06:30:38.597 [category-service] [main] INFO  c.a.n.p.auth.spi.client.ClientAuthPluginManager - [] [ClientAuthPluginManager] Load ClientAuthService com.alibaba.nacos.client.auth.ram.RamClientAuthServiceImpl success.
2023-06-27 06:30:38.740 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [RpcClientFactory] create a new rpc client of 87d3b9e9-724c-4edf-8fd7-ad22c5218bb9
2023-06-27 06:30:38.740 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] RpcClient init label, labels = {module=naming, source=sdk}
2023-06-27 06:30:38.744 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] RpcClient init, ServerListFactory = com.alibaba.nacos.client.naming.core.ServerListManager
2023-06-27 06:30:38.744 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] Registry connection listener to current client:com.alibaba.nacos.client.naming.remote.gprc.redo.NamingGrpcRedoService
2023-06-27 06:30:38.744 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] Register server push request handler:com.alibaba.nacos.client.naming.remote.gprc.NamingPushRequestHandler
2023-06-27 06:30:38.745 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] Try to connect to server on start up, server: {serverIp = 'juru.nacos.jrmall.cn', server main port = 80}
2023-06-27 06:30:39.106 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] Success to connect to server [juru.nacos.jrmall.cn:80] on start up, connectionId = 1687847438992_172.2.1.6_60904
2023-06-27 06:30:39.106 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] Register server push request handler:com.alibaba.nacos.common.remote.client.RpcClient$ConnectResetRequestHandler
2023-06-27 06:30:39.106 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] Notify connected event to listeners.
2023-06-27 06:30:39.106 [category-service] [main] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] Register server push request handler:com.alibaba.nacos.common.remote.client.RpcClient$$Lambda$294/2008966511
2023-06-27 06:30:39.106 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.naming - [] Grpc connection connect
2023-06-27 06:30:39.166 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [fixed-dubbo-prod-juru.nacos.jrmall.cn_80] [subscribe] category-service.migration+DUBBO_SERVICEDISCOVERY_MIGRATION+dubbo-prod
2023-06-27 06:30:39.166 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.CacheData - [] [fixed-dubbo-prod-juru.nacos.jrmall.cn_80] [add-listener] ok, tenant=dubbo-prod, dataId=category-service.migration, group=DUBBO_SERVICEDISCOVERY_MIGRATION, cnt=1
2023-06-27 06:30:39.273 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [fixed-dubbo-prod-juru.nacos.jrmall.cn_80] [subscribe] org.apache.dubbo.metadata.MetadataService:1.0.0:category-service.configurators+dubbo+dubbo-prod
2023-06-27 06:30:39.273 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.CacheData - [] [fixed-dubbo-prod-juru.nacos.jrmall.cn_80] [add-listener] ok, tenant=dubbo-prod, dataId=org.apache.dubbo.metadata.MetadataService:1.0.0:category-service.configurators, group=dubbo, cnt=1
2023-06-27 06:30:39.357 [category-service] [DubboSaveMetadataReport-thread-1] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [fixed-dubbo-metadata-prod-juru.nacos.jrmall.cn_80] [publish-single] ok, dataId=com.jrmall.cate.service.CategoryServiceI:::provider:category-service, group=dubbo, tenant=dubbo-metadata-prod, config={"annotations":[],"canonicalName":"com.jrmall.cate.service.CategoryServiceI","codeSource":"jar:file:...
2023-06-27 06:30:39.427 [category-service] [main] INFO  com.alibaba.nacos.client.naming - [] [REGISTER-SERVICE] dubbo-prod registering service category-service with instance Instance{instanceId='null', ip='172.2.2.10', port=20880, weight=1.0, healthy=true, enabled=true, ephemeral=true, clusterName='null', serviceName='category-service', metadata={dubbo.endpoints=[{"port":20880,"protocol":"dubbo"}], dubbo.metadata-service.url-params={"connections":"1","timeout":"10000","version":"1.0.0","dubbo":"2.0.2","release":"3.1.7","side":"provider","port":"20880","protocol":"dubbo"}, dubbo.metadata.revision=974aa03938b53ab835ae070df94c8b03, dubbo.metadata.storage-type=local, timestamp=1687847438230}}
2023-06-27 06:30:39.444 [category-service] [main] INFO  com.jrmall.cate.CategoryServiceApplication - [] Started CategoryServiceApplication in 12.201 seconds (JVM running for 12.868)
2023-06-27 06:30:39.453 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [fixed-prod-juru.nacos.jrmall.cn_80] [subscribe] category-service.properties+DEFAULT_GROUP+prod
2023-06-27 06:30:39.453 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.CacheData - [] [fixed-prod-juru.nacos.jrmall.cn_80] [add-listener] ok, tenant=prod, dataId=category-service.properties, group=DEFAULT_GROUP, cnt=1
2023-06-27 06:30:39.453 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [fixed-prod-juru.nacos.jrmall.cn_80] [subscribe] category-service+DEFAULT_GROUP+prod
2023-06-27 06:30:39.453 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.CacheData - [] [fixed-prod-juru.nacos.jrmall.cn_80] [add-listener] ok, tenant=prod, dataId=category-service, group=DEFAULT_GROUP, cnt=1
2023-06-27 06:30:39.454 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.ClientWorker - [] [fixed-prod-juru.nacos.jrmall.cn_80] [subscribe] category-service-prod.properties+DEFAULT_GROUP+prod
2023-06-27 06:30:39.454 [category-service] [main] INFO  com.alibaba.nacos.client.config.impl.CacheData - [] [fixed-prod-juru.nacos.jrmall.cn_80] [add-listener] ok, tenant=prod, dataId=category-service-prod.properties, group=DEFAULT_GROUP, cnt=1
2023-06-27 06:30:47.711 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Server healthy check fail, currentConnection = 1687847434811_172.2.1.6_37568
2023-06-27 06:30:47.712 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-27 06:30:50.718 [category-service] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 120098 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@2bf1a5c4[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5c42730e, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1940f6e2, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f3f645c}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:192)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:302)
        at com.alibaba.nacos.common.remote.client.RpcClient.reconnect(RpcClient.java:521)
        at com.alibaba.nacos.common.remote.client.RpcClient.lambda$start$2(RpcClient.java:369)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-06-27 06:30:51.746 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] Server healthy check fail, currentConnection = 1687847438992_172.2.1.6_60904
2023-06-27 06:30:51.746 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-27 06:30:51.939 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Server healthy check fail, currentConnection = 1687847433533_172.2.1.6_56312
2023-06-27 06:30:51.940 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-27 06:30:52.174 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [8f03cfe7-dd05-4755-b7fa-b074b804c329_config-0] Server healthy check fail, currentConnection = 1687847430053_172.2.1.6_40928
2023-06-27 06:30:52.174 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [8f03cfe7-dd05-4755-b7fa-b074b804c329_config-0] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-06-27 06:30:53.829 [category-service] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 109200 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@1c901f68[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5c42730e, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1940f6e2, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f3f645c}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:192)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:302)
        at com.alibaba.nacos.common.remote.client.RpcClient.reconnect(RpcClient.java:521)
        at com.alibaba.nacos.common.remote.client.RpcClient.lambda$start$2(RpcClient.java:369)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-06-27 06:30:53.830 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Fail to connect server, after trying 1 times, last try server is {serverIp = 'juru.nacos.jrmall.cn', server main port = 80}, error = unknown
2023-06-27 06:30:54.750 [category-service] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 103343 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@38849f1c[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5c42730e, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1940f6e2, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f3f645c}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:192)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:302)
        at com.alibaba.nacos.common.remote.client.RpcClient.reconnect(RpcClient.java:521)
        at com.alibaba.nacos.common.remote.client.RpcClient.lambda$start$2(RpcClient.java:369)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-06-27 06:30:54.942 [category-service] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 81665 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@7ca8a966[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5c42730e, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1940f6e2, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f3f645c}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:192)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:302)
        at com.alibaba.nacos.common.remote.client.RpcClient.reconnect(RpcClient.java:521)
        at com.alibaba.nacos.common.remote.client.RpcClient.lambda$start$2(RpcClient.java:369)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-06-27 06:30:55.177 [category-service] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 83735 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@5d445e53[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5c42730e, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1940f6e2, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f3f645c}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:192)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:302)
        at com.alibaba.nacos.common.remote.client.RpcClient.reconnect(RpcClient.java:521)
        at com.alibaba.nacos.common.remote.client.RpcClient.lambda$start$2(RpcClient.java:369)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-06-27 06:30:57.035 [category-service] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 118654 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@3c7c0bf2[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5c42730e, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1940f6e2, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f3f645c}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:192)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:302)
        at com.alibaba.nacos.common.remote.client.RpcClient.reconnect(RpcClient.java:521)
        at com.alibaba.nacos.common.remote.client.RpcClient.lambda$start$2(RpcClient.java:369)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-06-27 06:30:57.036 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [3379f680-d06e-4a55-8c45-5e5c38e5d2d0_config-0] Fail to connect server, after trying 2 times, last try server is {serverIp = 'juru.nacos.jrmall.cn', server main port = 80}, error = unknown
2023-06-27 06:30:57.855 [category-service] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 103133 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@33f0a7a0[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5c42730e, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1940f6e2, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f3f645c}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:192)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:302)
        at com.alibaba.nacos.common.remote.client.RpcClient.reconnect(RpcClient.java:521)
        at com.alibaba.nacos.common.remote.client.RpcClient.lambda$start$2(RpcClient.java:369)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-06-27 06:30:57.856 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [87d3b9e9-724c-4edf-8fd7-ad22c5218bb9] Fail to connect server, after trying 1 times, last try server is {serverIp = 'juru.nacos.jrmall.cn', server main port = 80}, error = unknown
2023-06-27 06:30:58.046 [category-service] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 110264 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@5703423a[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5c42730e, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1940f6e2, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f3f645c}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:192)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:302)
        at com.alibaba.nacos.common.remote.client.RpcClient.reconnect(RpcClient.java:521)
        at com.alibaba.nacos.common.remote.client.RpcClient.lambda$start$2(RpcClient.java:369)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-06-27 06:30:58.047 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [1c3880d3-b203-4658-8a44-d9b14971943a_config-0] Fail to connect server, after trying 1 times, last try server is {serverIp = 'juru.nacos.jrmall.cn', server main port = 80}, error = unknown
2023-06-27 06:30:58.282 [category-service] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 106135 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@3301b448[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5c42730e, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1940f6e2, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f3f645c}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.serverCheck(GrpcClient.java:192)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcClient.connectToServer(GrpcClient.java:302)
        at com.alibaba.nacos.common.remote.client.RpcClient.reconnect(RpcClient.java:521)
        at com.alibaba.nacos.common.remote.client.RpcClient.lambda$start$2(RpcClient.java:369)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2023-06-27 06:30:58.282 [category-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [8f03cfe7-dd05-4755-b7fa-b074b804c329_config-0] Fail to connect server, after trying 1 times, last try server is {serverIp = 'juru.nacos.jrmall.cn', server main port = 80}, error = unknown
2023-06-27 06:30:59.171 [category-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-28] WARN  c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687847430053_172.2.1.6_40928]Ignore error event,isRunning:false,isAbandon=false
2023-06-27 06:30:59.171 [category-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-34] WARN  c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687847433533_172.2.1.6_56312]Ignore error event,isRunning:false,isAbandon=false
2023-06-27 06:30:59.197 [category-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-12] WARN  c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687847434811_172.2.1.6_37568]Ignore error event,isRunning:false,isAbandon=false
2023-06-27 06:30:59.435 [category-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-10] WARN  c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1687847438992_172.2.1.6_60904]Ignore error event,isRunning:false,isAbandon=false
2023-06-27 06:31:00.340 [category-service] [com.alibaba.nacos.client.remote.worker] ERROR c.a.nacos.common.remote.client.grpc.GrpcClient - [] Server check fail, please check server juru.nacos.jrmall.cn ,port 1080 is available , error ={}
java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 112500 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@13ba8d50[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@5c42730e, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@1940f6e2, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@1f3f645c}}}}}]]

nginx容器IP：172.2.1.6
nginx配置：

user  root;
worker_processes auto;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
worker_rlimit_nofile 51200;

events
    {
        use epoll;
        worker_connections 51200;
        multi_accept on;
    }
stream {
  upstream nacos.grpc_server {
        server 172.2.1.3:9848;
        server 172.2.1.4:9848;
        server 172.2.1.5:9848;
    }

  server {
        listen 1080 so_keepalive=on;
        proxy_responses 1;
        proxy_timeout 20s;
        proxy_pass nacos.grpc_server;
    }
  upstream nacos.grpc_server.dev {
        server 172.16.85.105:9848;
        server 172.16.85.108:9848;
        server 172.16.85.108:9858;
    }

  server {
        listen 1090 so_keepalive=on;
        proxy_responses 1;
        proxy_timeout 20s;
        proxy_pass nacos.grpc_server.dev;
    }
}

http
    {
    include       mime.types;
        #include luawaf.conf;

    default_type  application/octet-stream;

    server_names_hash_bucket_size 512;
    client_header_buffer_size 32k;
    large_client_header_buffers 4 32k;
    client_max_body_size 300m;

    sendfile   on;
    tcp_nopush on;

    keepalive_timeout 60;

    tcp_nodelay on;

    fastcgi_connect_timeout 300;
    fastcgi_send_timeout 300;
    fastcgi_read_timeout 300;
    fastcgi_buffer_size 64k;
    fastcgi_buffers 4 64k;
    fastcgi_busy_buffers_size 128k;
    fastcgi_temp_file_write_size 256k;
        fastcgi_intercept_errors on;

    gzip on;
    gzip_min_length  1k;
    gzip_buffers     4 16k;
    gzip_http_version 1.1;
    gzip_comp_level 2;
    gzip_types     text/plain application/javascript application/x-javascript text/javascript text/css application/xml;
    gzip_vary on;
    gzip_proxied   expired no-cache no-store private auth;
    gzip_disable   "MSIE [1-6]\.";

    limit_conn_zone $binary_remote_addr zone=perip:10m;
        limit_conn_zone $server_name zone=perserver:10m;

    server_tokens off;
    access_log off;

    #include  /etc/nginx/conf.d/*.conf;

   upstream admin.jrmall.cn {
        server 172.2.1.12:9000;
        server 172.2.2.12:9000;
   }
   upstream nacos.jrmall.cn {
        server 172.2.1.3:8848;
        server 172.2.1.4:8848;
        server 172.2.1.5:8848;
   }
   upstream nacos-dev.jrmall.cn {
        server 172.16.85.105:8848;
        server 172.16.85.108:8848;
        server 172.16.85.108:8858;
   }

   server
        {
            listen 80;
            server_name juru.new.jrmall.cn;
            root  /usr/share/nginx/html;

            #禁止访问的文件或目录
            location ~ ^/(\.user.ini|\.htaccess|\.git|\.svn|\.project|LICENSE|README.md)
            {
                return 404;
            }

            location / {
                proxy_redirect off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass http://admin.jrmall.cn;
            }

            location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css|svg)$ {
              proxy_set_header   Host             $host;
              proxy_set_header   X-Real-IP        $remote_addr;
              proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
              proxy_pass http://admin.jrmall.cn;
            }


            location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$
            {
                expires      30d;
                error_log /dev/null;
                access_log /dev/null;
            }

            location ~ .*\.(js|css)?$
            {
                expires      12h;
                error_log /dev/null;
                access_log /dev/null;
            }
             error_page  500 502 503 504  /50x.html;

            location = /50x.html {
                root  /usr/share/nginx/html;
            }
            access_log  /var/log/nginx/access.log;
        }
   server
        {
            listen 80;
            server_name juru.nacos.jrmall.cn;
            root  /usr/share/nginx/html;

            #禁止访问的文件或目录
            location ~ ^/(\.user.ini|\.htaccess|\.git|\.svn|\.project|LICENSE|README.md)
            {
                return 404;
            }

            location / {
                proxy_redirect off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass http://nacos.jrmall.cn;
            }

            location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css|svg)$ {
              proxy_set_header   Host             $host;
              proxy_set_header   X-Real-IP        $remote_addr;
              proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
              proxy_pass http://nacos.jrmall.cn;
            }


            location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$
            {
                expires      30d;
                error_log /dev/null;
                access_log /dev/null;
            }

            location ~ .*\.(js|css)?$
            {
                expires      12h;
                error_log /dev/null;
                access_log /dev/null;
            }
             error_page  500 502 503 504  /50x.html;

            location = /50x.html {
                root  /usr/share/nginx/html;
            }
            access_log  /var/log/nginx/access.log;
        }
    server
        {
            listen 90;
            server_name juru.nacos-dev.jrmall.cn;
            root  /usr/share/nginx/html;

            #禁止访问的文件或目录
            location ~ ^/(\.user.ini|\.htaccess|\.git|\.svn|\.project|LICENSE|README.md)
            {
                return 404;
            }

            location / {
                proxy_redirect off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_pass http://nacos-dev.jrmall.cn;
            }

            location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css|svg)$ {
              proxy_set_header   Host             $host;
              proxy_set_header   X-Real-IP        $remote_addr;
              proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
              proxy_pass http://nacos.jrmall.cn;
            }


            location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$
            {
                expires      30d;
                error_log /dev/null;
                access_log /dev/null;
            }

            location ~ .*\.(js|css)?$
            {
                expires      12h;
                error_log /dev/null;
                access_log /dev/null;
            }
             error_page  500 502 503 504  /50x.html;

            location = /50x.html {
                root  /usr/share/nginx/html;
            }
            access_log  /var/log/nginx/access.log;
        }
}

@KomachiSion
Copy link
Collaborator

Connection refused: juru.nacos.jrmall.cn/172.2.1.6:1080

说明1080端口没有启动监听

@KomachiSion
Copy link
Collaborator

或者是1080映射的后端服务没有

@KomachiSion KomachiSion added the kind/question Category issues related to questions or problems label Jun 30, 2023
@lidaoyang
Copy link
Contributor Author

或者是1080映射的后端服务没有

是NGINX映射有问题？
image

@cnry1
Copy link

cnry1 commented Jun 30, 2023 via email

@lidaoyang
Copy link
Contributor Author

Reply to this email directly, view it on GitHub, or unsubscribe.

我用主机IP telnet 1080是通的额

image

@KomachiSion
Copy link
Collaborator

我感觉是你对你自己环境弄的很混乱，最好梳理一下，分开部署和配置。

@lidaoyang
Copy link
Contributor Author

我感觉是你对你自己环境弄的很混乱，最好梳理一下，分开部署和配置。

现在没有报grpc 连接错误了，不过在更新服务的时候等待超时了，接口调用没有问题，可以调通。
这个更新服务超时是什么原因啊？8848，9848，9849 端口都开了

2023-07-03 19:52:18.456 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-83] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Ack server push request, request = NotifySubscriberRequest, requestId = 699
2023-07-03 19:52:33.565 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-87] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Receive server push request, request = NotifySubscriberRequest, requestId = 709
2023-07-03 19:52:33.566 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-87] INFO  com.alibaba.nacos.client.naming - [] removed ips(1) service: DEFAULT_GROUP@@category-service -> [{"ip":"172.2.1.10","port":20880,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@category-service","metadata":{"dubbo.endpoints":"[{\"port\":20880,\"protocol\":\"dubbo\"}]","dubbo.metadata.revision":"01764b50e7c08ca356552ca71a9e9abe","dubbo.metadata.storage-type":"remote","timestamp":"1688384967104"},"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"ipDeleteTimeout":30000}]
2023-07-03 19:52:33.566 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-87] INFO  com.alibaba.nacos.client.naming - [] current ips:(1) service: DEFAULT_GROUP@@category-service -> [{"ip":"172.2.2.10","port":20880,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@category-service","metadata":{"dubbo.endpoints":"[{\"port\":20880,\"protocol\":\"dubbo\"}]","dubbo.metadata.revision":"01764b50e7c08ca356552ca71a9e9abe","dubbo.metadata.storage-type":"remote","timestamp":"1688288286527"},"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"ipDeleteTimeout":30000}]
2023-07-03 19:52:33.566 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-87] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Ack server push request, request = NotifySubscriberRequest, requestId = 709
2023-07-03 19:52:38.485 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-88] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Receive server push request, request = NotifySubscriberRequest, requestId = 710
2023-07-03 19:52:38.486 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-88] INFO  com.alibaba.nacos.client.naming - [] new ips(1) service: DEFAULT_GROUP@@category-service -> [{"ip":"172.2.1.10","port":20880,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@category-service","metadata":{"dubbo.endpoints":"[{\"port\":20880,\"protocol\":\"dubbo\"}]","dubbo.metadata.revision":"01764b50e7c08ca356552ca71a9e9abe","dubbo.metadata.storage-type":"remote","timestamp":"1688384967104"},"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"ipDeleteTimeout":30000}]
2023-07-03 19:52:38.486 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-88] INFO  com.alibaba.nacos.client.naming - [] current ips:(2) service: DEFAULT_GROUP@@category-service -> [{"ip":"172.2.2.10","port":20880,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@category-service","metadata":{"dubbo.endpoints":"[{\"port\":20880,\"protocol\":\"dubbo\"}]","dubbo.metadata.revision":"01764b50e7c08ca356552ca71a9e9abe","dubbo.metadata.storage-type":"remote","timestamp":"1688288286527"},"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"ipDeleteTimeout":30000},{"ip":"172.2.1.10","port":20880,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@category-service","metadata":{"dubbo.endpoints":"[{\"port\":20880,\"protocol\":\"dubbo\"}]","dubbo.metadata.revision":"01764b50e7c08ca356552ca71a9e9abe","dubbo.metadata.storage-type":"remote","timestamp":"1688384967104"},"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"ipDeleteTimeout":30000}]
2023-07-03 19:52:38.487 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-88] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Ack server push request, request = NotifySubscriberRequest, requestId = 710
2023-07-03 19:52:45.160 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-90] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Receive server push request, request = NotifySubscriberRequest, requestId = 711
2023-07-03 19:52:45.161 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-90] INFO  com.alibaba.nacos.client.naming - [] removed ips(1) service: DEFAULT_GROUP@@category-service -> [{"ip":"172.2.2.10","port":20880,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@category-service","metadata":{"dubbo.endpoints":"[{\"port\":20880,\"protocol\":\"dubbo\"}]","dubbo.metadata.revision":"01764b50e7c08ca356552ca71a9e9abe","dubbo.metadata.storage-type":"remote","timestamp":"1688288286527"},"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"ipDeleteTimeout":30000}]
2023-07-03 19:52:45.161 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-90] INFO  com.alibaba.nacos.client.naming - [] current ips:(1) service: DEFAULT_GROUP@@category-service -> [{"ip":"172.2.1.10","port":20880,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@category-service","metadata":{"dubbo.endpoints":"[{\"port\":20880,\"protocol\":\"dubbo\"}]","dubbo.metadata.revision":"01764b50e7c08ca356552ca71a9e9abe","dubbo.metadata.storage-type":"remote","timestamp":"1688384967104"},"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"ipDeleteTimeout":30000}]
2023-07-03 19:52:45.161 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-90] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Ack server push request, request = NotifySubscriberRequest, requestId = 711
2023-07-03 19:52:58.775 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-93] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Receive server push request, request = NotifySubscriberRequest, requestId = 712
2023-07-03 19:52:58.775 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-93] INFO  com.alibaba.nacos.client.naming - [] new ips(1) service: DEFAULT_GROUP@@category-service -> [{"ip":"172.2.2.10","port":20880,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@category-service","metadata":{"dubbo.endpoints":"[{\"port\":20880,\"protocol\":\"dubbo\"}]","dubbo.metadata.revision":"01764b50e7c08ca356552ca71a9e9abe","dubbo.metadata.storage-type":"remote","timestamp":"1688385177141"},"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"ipDeleteTimeout":30000}]
2023-07-03 19:52:58.775 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-93] INFO  com.alibaba.nacos.client.naming - [] current ips:(2) service: DEFAULT_GROUP@@category-service -> [{"ip":"172.2.2.10","port":20880,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@category-service","metadata":{"dubbo.endpoints":"[{\"port\":20880,\"protocol\":\"dubbo\"}]","dubbo.metadata.revision":"01764b50e7c08ca356552ca71a9e9abe","dubbo.metadata.storage-type":"remote","timestamp":"1688385177141"},"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"ipDeleteTimeout":30000},{"ip":"172.2.1.10","port":20880,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@category-service","metadata":{"dubbo.endpoints":"[{\"port\":20880,\"protocol\":\"dubbo\"}]","dubbo.metadata.revision":"01764b50e7c08ca356552ca71a9e9abe","dubbo.metadata.storage-type":"remote","timestamp":"1688384967104"},"instanceHeartBeatTimeOut":15000,"instanceHeartBeatInterval":5000,"ipDeleteTimeout":30000}]
2023-07-03 19:52:58.776 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-93] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Ack server push request, request = NotifySubscriberRequest, requestId = 712
2023-07-03 19:55:20.156 [system-service] [DubboServerHandler-172.2.1.9:20880-thread-11] DEBUG com.jrmall.system.dao.SystemAdminDao.selectList - [] ==>  Preparing: SELECT id,account,pwd,real_name,roles,status FROM jr_system_admin WHERE (type = ? AND account = ?)
2023-07-03 19:55:20.336 [system-service] [DubboServerHandler-172.2.1.9:20880-thread-11] DEBUG com.jrmall.system.dao.SystemAdminDao.selectList - [] ==> Parameters: PL_ADMIN(String), 15158152796(String)
2023-07-03 19:55:20.360 [system-service] [DubboServerHandler-172.2.1.9:20880-thread-11] DEBUG com.jrmall.system.dao.SystemAdminDao.selectList - [] <==      Total: 1
2023-07-03 19:55:20.376 [system-service] [DubboServerHandler-172.2.1.9:20880-thread-11] INFO  com.jrmall.trace.filter.rpc.ProviderRpcFilter - [7752b3dd8a1c4e79ba34cbb8285f2b33] 进入 Provider RPC Filter traceId:7752b3dd8a1c4e79ba34cbb8285f2b33,methodName:getUserBserName
2023-07-03 20:49:39.691 [system-service] [rpcDispatch_RMROLE_1_1_24] INFO  i.s.core.rpc.processor.client.RmUndoLogProcessor - [] rm handle undo log process:UndoLogDeleteRequest{resourceId='jdbc:mysql://172.2.1.2:3306/shop_new', saveDays=7, branchType=AT}
2023-07-03 23:10:25.995 [system-service] [com.alibaba.nacos.client.naming.updater.6] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ServiceQueryRequest{headers={}, requestId='null'}, retryTimes = 0, errorMessage = java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 146612 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@32245773[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
2023-07-03 23:10:26.001 [system-service] [com.alibaba.nacos.client.naming.updater.6] WARN  com.alibaba.nacos.client.naming - [] [NA] failed to update serviceName: DEFAULT_GROUP@@category-service
com.alibaba.nacos.api.exception.NacosException: Request nacos server failed: 
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:288)
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.queryInstancesOfService(NamingGrpcClientProxy.java:168)
        at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.queryInstancesOfService(NamingClientProxyDelegate.java:110)
        at com.alibaba.nacos.client.naming.core.ServiceInfoUpdateService$UpdateTask.run(ServiceInfoUpdateService.java:185)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.nacos.api.exception.NacosException: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 146612 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@32245773[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:78)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:653)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:631)
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:278)
        ... 10 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 146612 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@32245773[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:76)
        ... 13 common frames omitted
2023-07-03 23:10:26.038 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Server check success, currentServer is juru.nacos.jrmall.cn:80 
2023-07-03 23:11:25.520 [system-service] [com.alibaba.nacos.client.naming.updater.3] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ServiceQueryRequest{headers={}, requestId='null'}, retryTimes = 0, errorMessage = java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 52056 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@505f002d[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
2023-07-03 23:11:25.521 [system-service] [com.alibaba.nacos.client.naming.updater.3] WARN  com.alibaba.nacos.client.naming - [] [NA] failed to update serviceName: SEATA_GROUP@@seata-server
com.alibaba.nacos.api.exception.NacosException: Request nacos server failed: 
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:288)
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.queryInstancesOfService(NamingGrpcClientProxy.java:168)
        at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.queryInstancesOfService(NamingClientProxyDelegate.java:110)
        at com.alibaba.nacos.client.naming.core.ServiceInfoUpdateService$UpdateTask.run(ServiceInfoUpdateService.java:185)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.nacos.api.exception.NacosException: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 52056 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@505f002d[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:78)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:653)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:631)
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:278)
        ... 10 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 52056 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@505f002d[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:76)
        ... 13 common frames omitted
2023-07-03 23:11:25.547 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [19501498-2a6b-441e-8e11-9f6a42ef07ef] Server check success, currentServer is juru.nacos.jrmall.cn:80 
2023-07-03 23:11:30.522 [system-service] [com.alibaba.nacos.client.naming.updater.3] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = ServiceQueryRequest{headers={}, requestId='null'}, retryTimes = 0, errorMessage = java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 78390 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@29384066[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
2023-07-03 23:11:30.522 [system-service] [com.alibaba.nacos.client.naming.updater.3] WARN  com.alibaba.nacos.client.naming - [] [NA] failed to update serviceName: SEATA_GROUP@@seata-server
com.alibaba.nacos.api.exception.NacosException: Request nacos server failed: 
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:288)
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.queryInstancesOfService(NamingGrpcClientProxy.java:168)
        at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.queryInstancesOfService(NamingClientProxyDelegate.java:110)
        at com.alibaba.nacos.client.naming.core.ServiceInfoUpdateService$UpdateTask.run(ServiceInfoUpdateService.java:185)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.nacos.api.exception.NacosException: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 78390 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@29384066[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:78)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:653)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:631)
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:278)
        ... 10 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 78390 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@29384066[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:76)
        ... 13 common frames omitted
2023-07-03 23:11:30.860 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [19501498-2a6b-441e-8e11-9f6a42ef07ef] Server check success, currentServer is juru.nacos.jrmall.cn:80 
2023-07-04 01:33:56.853 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Server healthy check fail, currentConnection = 1688384934471_172.2.1.6_42652
2023-07-04 01:33:56.857 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-07-04 01:33:57.319 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Success to connect a server [juru.nacos.jrmall.cn:80], connectionId = 1688405638025_172.2.1.6_35620
2023-07-04 01:33:57.319 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Abandon prev connection, server is juru.nacos.jrmall.cn:80, connectionId is 1688384934471_172.2.1.6_42652
2023-07-04 01:33:57.319 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] Close current connection 1688384934471_172.2.1.6_42652
2023-07-04 01:33:57.324 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Notify disconnected event to listeners
2023-07-04 01:33:57.325 [system-service] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] Grpc connection disconnect, mark to redo
2023-07-04 01:33:57.328 [system-service] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] mark to redo completed
2023-07-04 01:33:57.328 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Notify connected event to listeners.
2023-07-04 01:33:57.328 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.naming - [] Grpc connection connect
2023-07-04 01:33:57.329 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-4821] WARN  c.a.nacos.common.remote.client.grpc.GrpcClient - [] [1688384934471_172.2.1.6_42652]Ignore error event,isRunning:true,isAbandon=true
2023-07-04 01:34:00.120 [system-service] [com.alibaba.nacos.client.naming.grpc.redo.0] INFO  com.alibaba.nacos.client.naming - [] Redo instance operation REGISTER for DEFAULT_GROUP@@system-service
2023-07-04 01:34:03.122 [system-service] [com.alibaba.nacos.client.naming.grpc.redo.0] ERROR com.alibaba.nacos.common.remote.client - [] Send request fail, request = InstanceRequest{headers={}, requestId='null'}, retryTimes = 0, errorMessage = java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 61341 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@361b32e8[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
2023-07-04 01:34:03.124 [system-service] [com.alibaba.nacos.client.naming.grpc.redo.0] ERROR com.alibaba.nacos.client.naming - [] Redo instance operation REGISTER for DEFAULT_GROUP@@system-service failed. 
com.alibaba.nacos.api.exception.NacosException: Request nacos server failed: 
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:288)
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.doRegisterService(NamingGrpcClientProxy.java:128)
        at com.alibaba.nacos.client.naming.remote.gprc.redo.RedoScheduledTask.redoForInstance(RedoScheduledTask.java:78)
        at com.alibaba.nacos.client.naming.remote.gprc.redo.RedoScheduledTask.redoForInstances(RedoScheduledTask.java:60)
        at com.alibaba.nacos.client.naming.remote.gprc.redo.RedoScheduledTask.run(RedoScheduledTask.java:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.nacos.api.exception.NacosException: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 61341 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@361b32e8[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:78)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:653)
        at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:631)
        at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:278)
        ... 11 common frames omitted
Caused by: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 61341 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@361b32e8[status=PENDING, info=[GrpcFuture{clientCall={delegate={delegate=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@68e830d5, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@23802823, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@330388df}}}}}]]
        at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:508)
        at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:76)
        ... 14 common frames omitted
2023-07-04 01:34:03.124 [system-service] [com.alibaba.nacos.client.naming.grpc.redo.0] INFO  com.alibaba.nacos.client.naming - [] Redo subscriber operation REGISTER for DEFAULT_GROUP@@category-service#
2023-07-04 01:34:03.245 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Server check success, currentServer is juru.nacos.jrmall.cn:80 
2023-07-04 01:34:06.125 [system-service] [com.alibaba.nacos.client.naming.grpc.redo.0] INFO  com.alibaba.nacos.client.naming - [] Redo instance operation REGISTER for DEFAULT_GROUP@@system-service
2023-07-04 01:34:08.806 [system-service] [com.alibaba.nacos.client.naming.grpc.redo.0] INFO  com.alibaba.nacos.client.naming - [] Redo subscriber operation REGISTER for DEFAULT_GROUP@@category-service#
2023-07-04 01:34:09.376 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-4830] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Receive server push request, request = NotifySubscriberRequest, requestId = 726
2023-07-04 01:34:09.376 [system-service] [nacos-grpc-client-executor-juru.nacos.jrmall.cn-4830] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Ack server push request, request = NotifySubscriberRequest, requestId = 726
2023-07-04 01:36:36.574 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Server healthy check fail, currentConnection = 1688405638025_172.2.1.6_35620
2023-07-04 01:36:36.575 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Try to reconnect to a new server, server is  not appointed, will choose a random server.
2023-07-04 01:36:36.687 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Success to connect a server [juru.nacos.jrmall.cn:80], connectionId = 1688405796581_172.2.1.6_58254
2023-07-04 01:36:36.687 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Abandon prev connection, server is juru.nacos.jrmall.cn:80, connectionId is 1688405638025_172.2.1.6_35620
2023-07-04 01:36:36.688 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] Close current connection 1688405638025_172.2.1.6_35620
2023-07-04 01:36:36.689 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Notify disconnected event to listeners
2023-07-04 01:36:36.689 [system-service] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] Grpc connection disconnect, mark to redo
2023-07-04 01:36:36.689 [system-service] [com.alibaba.nacos.client.remote.worker] WARN  com.alibaba.nacos.client.naming - [] mark to redo completed
2023-07-04 01:36:36.689 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.common.remote.client - [] [9b72406a-af2f-405e-b0e6-146cb4ef2380] Notify connected event to listeners.
2023-07-04 01:36:36.689 [system-service] [com.alibaba.nacos.client.remote.worker] INFO  com.alibaba.nacos.client.naming - [] Grpc connection connect

@KomachiSion
Copy link
Collaborator

超时无非3种情况：

  1. 数据包发出后被丢包（可能发起时被丢弃，可能服务端收到后，回包被丢弃），导致客户端无法收到回包。
  2. 服务端压力过大，响应不过来导致超时。
  3. 服务端正常响应，客户端压力过大，无资源处理回包导致超时。

具体那种原因可以看下服务端是否收到了请求并进行处理，或者抓网络包看服务端是否收到了包，是否回了包。

@lidaoyang
Copy link
Contributor Author

超时无非3种情况：

  1. 数据包发出后被丢包（可能发起时被丢弃，可能服务端收到后，回包被丢弃），导致客户端无法收到回包。
  2. 服务端压力过大，响应不过来导致超时。
  3. 服务端正常响应，客户端压力过大，无资源处理回包导致超时。

具体那种原因可以看下服务端是否收到了请求并进行处理，或者抓网络包看服务端是否收到了包，是否回了包。

好的，我在观察一下

@KomachiSion
Copy link
Collaborator

No more response from author, I think this is a env problem.

@KomachiSion KomachiSion closed this as not planned Won't fix, can't repro, duplicate, stale Jul 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/question Category issues related to questions or problems
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lidaoyang @cnry1 @KomachiSion @985492783