Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elastic 5.1, cat indices NPE, cat nodes master removed and readded #24696

Closed
sinsonglew opened this issue May 16, 2017 · 6 comments
Closed

elastic 5.1, cat indices NPE, cat nodes master removed and readded #24696

sinsonglew opened this issue May 16, 2017 · 6 comments

Comments

@sinsonglew
Copy link

sinsonglew commented May 16, 2017

Describe the feature:

Elasticsearch version: 5.1.1

Plugins installed: []

JVM version (java -version): 1.8_111

OS version (uname -a if on a Unix-like system): centOS 6.x

Description of the problem including expected versus actual behavior:

expectation:
cat indices list each index normally;
cat nodes show complete infomation

actually:
NPE/missed some node info, the master node included

Steps to reproduce:

  1. when cat nodes, not each node info got listed, some lack load and cpu info;
  2. came to node logs, NodeNotConnectedException appears and found many nodes got removed;

Provide logs (if relevant):
`[2017-05-16T11:36:10,005][WARN ][o.e.t.n.Netty3Transport ] [node75] exception caught on transport layer [[id: 0x32b26cc8, /10.07.24.75:59742 => 10.07.24.79/10.07.24.79:9371]], closing connection
java.lang.IllegalStateException: Message not fully read (response) for requestId [143219], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1@75cacb24], error [false]; resetting
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1257) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.netty3.Netty3MessageChannelHandler.messageReceived(Netty3MessageChannelHandler.java:73) ~[transport-netty3-5.1.1.jar:5.1.1]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
[2017-05-16T11:36:10,005][WARN ][o.e.t.n.Netty3Transport ] [node75] exception caught on transport layer [[id: 0x1ff141b8, /10.07.24.75:50015 => 10.07.24.72/10.07.24.72:9371]], closing connection
java.lang.IllegalStateException: Message not fully read (response) for requestId [143217], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1@38654ffd], error [false]; resetting
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1257) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.netty3.Netty3MessageChannelHandler.messageReceived(Netty3MessageChannelHandler.java:73) ~[transport-netty3-5.1.1.jar:5.1.1]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
[2017-05-16T11:36:10,007][INFO ][o.e.d.z.ZenDiscovery ] [node75] master_left [{node72}{C96zII7GQAywJY-Z15a-iQ}{480aO4cUTTWT53z6fI4CfA}{10.07.24.72}{10.07.24.72:9371}], reason [transport disconnected]
[2017-05-16T11:36:10,008][WARN ][o.e.d.z.ZenDiscovery ] [node75] master left (reason = transport disconnected), current nodes: nodes:
{node77}{_dn3sUWxRFGqfsFN1FQ_ag}{n2_JFL96RIy9AvJ-hchpTw}{10.07.24.77}{10.07.24.77:9371}
{node74}{bvG9NZuMS9Krrz9FCQnOsw}{-aXpRRNWRymTjbjRFwXD-A}{10.07.24.74}{10.07.24.74:9371}
{node79}{R7E5QswhQy6sSQt2DUKazQ}{Ic538WxUR42bUYQJY1pMJg}{10.07.24.79}{10.07.24.79:9371}
{node78}{FgfCdXu1R9qqmpyIRPf5LQ}{P9UVxmSkTX-cbhUaIaHRBA}{10.07.24.78}{10.07.24.78:9371}
{node22}{jn1MmeOKRnGnS-HiXAutOw}{WgYK9HHfR82Pp3sa10lkIA}{10.07.27.22}{10.07.27.22:9371}
{node70}{tJp2oOCyTXWb3XGnHACr9A}{rBwp8j4IQmW_cvQoZDLlYw}{10.07.24.70}{10.07.24.70:9371}
{node71}{g1XDb-cGTCqOVLgwL5dW7A}{mFJTjLRAQdOVSI1U_G5kcA}{10.07.24.71}{10.07.24.71:9371}
{node76}{VEJVuKCSREugsJeRGkbigA}{nDqh04oJTwuPle1pOKhSgw}{10.07.24.76}{10.07.24.76:9371}
{node75}{7kgChQzaRoWPeBF6r0nHxg}{dWGdAmlgQcCJ44mSoIUgsw}{10.07.24.75}{10.07.24.75:9371}, local
{node73}{GCSMRAArS8ik7aoOtQnxCA}{cqWXaEtARQGIOUXsb81hvQ}{10.07.24.73}{10.07.24.73:9371}
{node24}{qvz24kf1SJC6SIMgtdBm6Q}{ONKezttVS2WtrLNh98qq5Q}{10.07.27.24}{10.07.27.24:9371}
{node23}{EEqGk6EwSwKO76w1KrZn0A}{ktTICW1WTfGqDjyXI4SrlA}{10.07.27.23}{10.07.27.23:9371}

[2017-05-16T11:36:10,008][INFO ][o.e.c.s.ClusterService ] [node75] removed {{node72}{C96zII7GQAywJY-Z15a-iQ}{480aO4cUTTWT53z6fI4CfA}{10.07.24.72}{10.07.24.72:9371},}, reason: master_failed ({node72}{C96zII7GQAywJY-Z15a-iQ}{480aO4cUTTWT53z6fI4CfA}{10.07.24.72}{10.07.24.72:9371})
[2017-05-16T11:36:13,020][INFO ][o.e.c.s.ClusterService ] [node75] detected_master {node72}{C96zII7GQAywJY-Z15a-iQ}{480aO4cUTTWT53z6fI4CfA}{10.07.24.72}{10.07.24.72:9371}, added {{node72}{C96zII7GQAywJY-Z15a-iQ}{480aO4cUTTWT53z6fI4CfA}{10.07.24.72}{10.07.24.72:9371},}, reason: zen-disco-receive(from master [master {node72}{C96zII7GQAywJY-Z15a-iQ}{480aO4cUTTWT53z6fI4CfA}{10.07.24.72}{10.07.24.72:9371} committed version [1896]])
`

@danielmitterdorfer
Copy link
Member

Hi @sinsonglew! These events may be correlated but I do not see evidence that one causes the other. We reserve Github for bug reports and feature requests only. Please ask questions like these in the Elasticsearch forum instead. Thank you!

@liliguo2023
Copy link

请教个问题 #22189 你改成netty3之后是否引发了别的问题?

@sinsonglew
Copy link
Author

sinsonglew commented May 25, 2017 via email

@liliguo2023
Copy link

@sinsonglew 好的,你们是在云上面弄的es吗? 我在5.1 5.2 5.4都遇到了这个问题。。。。

@sinsonglew
Copy link
Author

sinsonglew commented May 25, 2017 via email

@liliguo2023
Copy link

我现在还不确定是哪里的问题,我在阿里云有两套环境,只有其中一套有问题, 我用同样的配置在我们内网虚拟机没有遇到这个问题, 我在去研究下,如果确定了就去在重开下那个issue, 感谢您的回复

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants