Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stats: _status with #shards >> queue capacity failing with BroadcastShardOperationFailedException #7916

Closed
alexbrasetvik opened this Issue Sep 29, 2014 · 2 comments

Comments

Projects
None yet
4 participants
@alexbrasetvik
Copy link
Member

alexbrasetvik commented Sep 29, 2014

Hi.

While /_status is deprecated and should be replaced with /_recovery, there seems to be a regression between 1.3.2 and 1.3.3 when a cluster has lots of shards.

The /_status-request fails with e.g:

"failures" : [ {
      "index" : "foo-0104",
      "shard" : 0,
      "reason" : "BroadcastShardOperationFailedException[[foo-0104][0] ]; nested: EsRejectedExecutionException[rejected execution (queue capacity 100) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@759c850c]; "
    } ]
  }

Elasticsearch logs

[2014-09-29 19:05:09,657][DEBUG][action.admin.indices.status] [Sputnik] [foo-0104][0], node[LWIGdaF4QdWwLEz2peydqQ], [P], s[STARTED]: failed to executed [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@746ef917]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 100) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@759c850c
    at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:166)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.start(TransportBroadcastOperationAction.java:150)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:70)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:46)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:65)
    at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:71)
    at org.elasticsearch.client.support.AbstractIndicesAdminClient.status(AbstractIndicesAdminClient.java:429)
    at org.elasticsearch.rest.action.admin.indices.status.RestIndicesStatusAction.handleRequest(RestIndicesStatusAction.java:62)
    at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:66)
    at org.elasticsearch.rest.RestController.executeHandler(RestController.java:177)
    at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:160)
    at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
    at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
    at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:301)
    at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:44)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

This reliably reproduces the issue with a clean install of 1.3.3:

  • Create 200 indexes, e.g. >>> for i in range(0, 200): requests.put('http://localhost:9200/foo-%04i' % i, data='{"index": {"number_of_shards": 1, "number_of_replicas": 0}}')
  • curl localhost:9200/_status

This, in turn, causes dashboards like Kopf and Head to break.

@jprante

This comment has been minimized.

Copy link
Contributor

jprante commented Sep 29, 2014

The cause of this is a change in the management thread pool, see #7318

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Sep 30, 2014

this has been fixed by 4d5f6a7

@s1monw s1monw closed this Sep 30, 2014

@clintongormley clintongormley changed the title _status with #shards >> queue capacity failing with BroadcastShardOperationFailedException Stats: _status with #shards >> queue capacity failing with BroadcastShardOperationFailedException Oct 1, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.