Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

部署redis-cluster时:ERR Unknown node **** #92

Closed
csspark opened this issue Aug 10, 2017 · 5 comments
Closed

部署redis-cluster时:ERR Unknown node **** #92

csspark opened this issue Aug 10, 2017 · 5 comments

Comments

@csspark
Copy link

csspark commented Aug 10, 2017

2017-08-10 17:12:49.467 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - runInstance-fallback : redis-cli -h ... -p 6386 shutdown
2017-08-10 17:12:49.507 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6381 isSingleNode
2017-08-10 17:12:49.580 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6382 isSingleNode
2017-08-10 17:12:49.636 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6383 isSingleNode
2017-08-10 17:12:49.699 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6384 isSingleNode
2017-08-10 17:12:49.764 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6385 isSingleNode
2017-08-10 17:12:49.818 {http-nio-9999-exec-1} WARN com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6386 isSingleNode
2017-08-10 17:12:52.389 {http-nio-9999-exec-1} ERROR com.sohu.cache.redis.impl.RedisDeployCenterImpl - ERR Unknown node 9168cd159e927816698db9effbe7f04c07fcc073
redis.clients.jedis.exceptions.JedisDataException: ERR Unknown node 9168cd159e927816698db9effbe7f04c07fcc073
at redis.clients.jedis.Protocol.processError(Protocol.java:118) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na]
at redis.clients.jedis.Protocol.process(Protocol.java:152) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na]
at redis.clients.jedis.Protocol.read(Protocol.java:206) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na]
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:282) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na]
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:208) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na]
at redis.clients.jedis.Jedis.clusterReplicate(Jedis.java:3343) ~[cachecloud-jedis-1.0-SNAPSHOT.jar:na]
at com.sohu.cache.redis.impl.RedisDeployCenterImpl$4.execute(RedisDeployCenterImpl.java:241) ~[classes/:na]
at com.sohu.cache.util.IdempotentConfirmer.run(IdempotentConfirmer.java:27) [classes/:na]
at com.sohu.cache.redis.impl.RedisDeployCenterImpl.startCluster(RedisDeployCenterImpl.java:257) [classes/:na]
at com.sohu.cache.redis.impl.RedisDeployCenterImpl.deployClusterInstance(RedisDeployCenterImpl.java:104) [classes/:na]
at com.sohu.cache.stats.app.impl.AppDeployCenterImpl.deployCluster(AppDeployCenterImpl.java:447) [classes/:na]
at com.sohu.cache.stats.app.impl.AppDeployCenterImpl.allocateResourceApp(AppDeployCenterImpl.java:334) [classes/:na]
at com.sohu.cache.web.controller.AppManageController.doAddAppDeploy(AppManageController.java:477) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_141]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_141]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_141]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:222) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:814) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:737) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:969) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:860) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:845) [spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:243) [spring-boot-actuator-1.3.1.RELEASE.jar:1.3.1.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111) [spring-boot-actuator-1.3.1.RELEASE.jar:1.3.1.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103) [spring-boot-actuator-1.3.1.RELEASE.jar:1.3.1.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500) [tomcat-embed-core-8.0.30.jar:8.0.30]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456) [tomcat-embed-core-8.0.30.jar:8.0.30]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.0.30.jar:8.0.30]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
......

2017-08-10 17:13:02.446 {http-nio-9999-exec-1} ERROR com.sohu.cache.redis.impl.RedisDeployCenterImpl - ...:6382 set replicate:{}
2017-08-10 17:13:02.446 {http-nio-9999-exec-1} ERROR com.sohu.cache.redis.impl.RedisDeployCenterImpl - startCluster create error!

@geekyijun
Copy link
Collaborator

新建集群时出现Unknown node,一般是cluster节点meet通信失败或者异步操作延迟导致,请在节点上执行cluster nodes看下这些Unknown 节点的nodeId是否已经存在。如果不存在请检查redis日志。通常是网络防火墙限制导致。

@xianwei
Copy link

xianwei commented Mar 7, 2019

新建集群时出现Unknown node,一般是cluster节点meet通信失败或者异步操作延迟导致,请在节点上执行cluster nodes看下这些Unknown 节点的nodeId是否已经存在。如果不存在请检查redis日志。通常是网络防火墙限制导致。

cluster nodes查看是有对应的nodeId的,但是还是包该错误

@elemman
Copy link

elemman commented Mar 26, 2020

我也遇到这个问题

  1. 防火墙已关闭
  2. 用cluster nodes也有对应nodeId
  3. 直接在机器上操作redis连接都是通的,请问这个有解决办法了没有?

@wuwo1952368901
Copy link

我也是这种情况,在meet时只有在自己的nodes.conf中有记录写进来,其他节点的都没有,请问这是正常的吗?

@githubname1024
Copy link
Collaborator

如遇到该问题,请确认如下信息,如检查后,仍有问题,可reopen或者重新提issue。

  1. 在redis实例上,执行 cluster nodes 确认是否有该nodeId;
  2. redis 实例间网络是否连通,是否被防火墙拦截;
  3. 如网络确认正常,可重试部署任务,查看是否能够部署成功。

另,如果网络存在延迟或部署的第一次报上述错误,可以通过查看最终是否部署成功。默认上述异常对应的命令存在重试机制,如果因为网络延迟,可能会报异常,但如重试成功,不影响最终部署。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants