New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal: Wait until engine has started up when acquiring searcher #7455

Closed
spinscale opened this Issue Aug 26, 2014 · 0 comments

Comments

Projects
None yet
2 participants
@spinscale
Member

spinscale commented Aug 26, 2014

This exception happened after a node restart and a delete-by-query hits the node immediately. Happened on 1.3.1

[2014-08-25 16:34:50,926][ERROR][index.engine.internal    ] [node_name] [2013_09][3] failed to acquire searcher, source delete_by_query
java.lang.NullPointerException
    at org.elasticsearch.index.engine.internal.InternalEngine.acquireSearcher(InternalEngine.java:694)
    at org.elasticsearch.index.shard.service.InternalIndexShard.acquireSearcher(InternalIndexShard.java:653)
    at org.elasticsearch.action.deletebyquery.TransportShardDeleteByQueryAction.shardOperationOnReplica(TransportShardDeleteByQueryAction.java:139)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:242)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:221)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
[2014-08-25 16:34:50,927][WARN ][index.engine.internal    ] [node_name] [2013_09][3] failed engine [deleteByQuery/shard failed on replica]
[2014-08-25 16:35:13,744][WARN ][cluster.action.shard     ] [node_name] [2013_09][3] sending failed shard for [2013_09][3], node[0Y7oOI64Qea6GCaSh3OtLw], [R], s[INITIALIZING], indexUUID [_na_], reason [engine failure, message [deleteByQuery/shard failed on replica][EngineException[[2013_09][3] failed to acquire searcher, source delete_by_query]; nested: NullPointerException; ]]

@spinscale spinscale added the bug label Aug 26, 2014

@s1monw s1monw added v2.0.0 labels Aug 26, 2014

@s1monw s1monw self-assigned this Aug 26, 2014

s1monw added a commit to s1monw/elasticsearch that referenced this issue Aug 26, 2014

[ENGINE] Wait until engine is started up when acquireing searcher
Today we have a small window where a searcher can be acquired but the
engine is in the state of starting up. This causes a NPE triggering a
shard failure if we are fast enough. This commit fixes this situation
gracefully.

Closes elastic#7455

s1monw added a commit that referenced this issue Aug 26, 2014

[ENGINE] Wait until engine is started up when acquireing searcher
Today we have a small window where a searcher can be acquired but the
engine is in the state of starting up. This causes a NPE triggering a
shard failure if we are fast enough. This commit fixes this situation
gracefully.

Closes #7455

s1monw added a commit that referenced this issue Aug 26, 2014

[ENGINE] Wait until engine is started up when acquireing searcher
Today we have a small window where a searcher can be acquired but the
engine is in the state of starting up. This causes a NPE triggering a
shard failure if we are fast enough. This commit fixes this situation
gracefully.

Closes #7455

@s1monw s1monw closed this in #7456 Aug 26, 2014

@spinscale spinscale changed the title from Node startup: NPE triggers failed shard to Engine: Wait until engine is started up when acquiring searcher Aug 27, 2014

@clintongormley clintongormley changed the title from Engine: Wait until engine is started up when acquiring searcher to Internal: Wait until engine has started up when acquiring searcher Sep 8, 2014

s1monw added a commit that referenced this issue Sep 8, 2014

[ENGINE] Wait until engine is started up when acquireing searcher
Today we have a small window where a searcher can be acquired but the
engine is in the state of starting up. This causes a NPE triggering a
shard failure if we are fast enough. This commit fixes this situation
gracefully.

Closes #7455

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

[ENGINE] Wait until engine is started up when acquireing searcher
Today we have a small window where a searcher can be acquired but the
engine is in the state of starting up. This causes a NPE triggering a
shard failure if we are fast enough. This commit fixes this situation
gracefully.

Closes elastic#7455
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment