-
Notifications
You must be signed in to change notification settings - Fork 24.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Node disconnection during index creation leaves behind unassigned shards #9323
Comments
Looks like this was caused by a socket connection issue which caused the master to disconnect the data node when it was trying to allocate replicas. As a result, the master lost the single copy it had of shard 0 and therefore also cannot allocate the replica. We opted to leave these shards in place to prevent any potential loss of data. |
What do you do to recover from this situation? I'm having trouble restarting the service on the node on which it failed. |
@emwalker once a primary shard is started it will accept indexing requests which is why ES can't automatically re-assign it once lost. Rather it waits for the node to re-join the cluster (and re-assign it there). If you know nothing was indexed or you don't mind the data loss, you can use the reroute API to force assign the primary somewhere else. Obviously it will be empty. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-reroute.html#cluster-reroute (see the |
@ppf2 What version of Elasticsearch was running when you experienced this issue? |
I believe it was 1.4.1 @jlmoody |
i am also facing the same problem during the type update org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping [Activity]) within 30s |
We are also facing this during PUT requests to update the mapping:- Noticing this happens particularly when there are relocating shards on the cluster. Elasticsearch.Net.ElasticsearchServerException: ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [documenthighlight]) within 30s Is this going to be fixed in a later version? |
@vineet85 what you describe sounds like a different issue. Can you open one with all the details filled in? |
We are also getting the same error: [2017-02-05 02:00:47,371][DEBUG][action.admin.indices.create] [Nuke - Squadron Supreme Member] [flowtrader-middletier-2017.02.05] failed to create I'm not able to see the solution provided here or am i missing something. ES version: 1.7.5 |
@ImdotnetJunkie 1.7.5 is no longer supported. Time to upgrade - so much has changed and been improved |
@clintongormley : Yeah, we are already on our way to upgrade and now we got another good reason to upgrade. Thanks |
It looks like if there is a node connection issue during index creation between the master node and a data node that it is trying to initialize a shard on, it will leave the primary and replica of the shard in an unassigned state and will not retry or try to allocate it somewhere else.
The text was updated successfully, but these errors were encountered: