Failure to recover shards after disk is full #15333
Labels
blocker
>bug
:Distributed/Engine
Anything around managing Lucene and the Translog in an open shard.
:Distributed/Recovery
Anything around constructing a new shard, either from a local or a remote source.
v2.0.2
v2.1.1
v2.2.0
v5.0.0-alpha1
Comments
|
maybe you can tell us what prevented you from starting up again? |
|
There were 15 indices on the node. Of those 15 indices, 9 indices had issues with their shards and ES status was red. Issue curl -XGET http://localhost:9200/_cat/shards command, listed 52 shards UNASSIGNED and 4 shards in INITIALIZING status. I issued reroute command (localhost:9200/_cluster/reroute) to move UNASSIGNED to force shard allocation. However, the shards that were in INITIALIZING status stay there. The CPU usage was 100% (8-cores busy) for more than 4 hours, before I gave up and started deleting all indices that were causing the problem. Data was about 50GB and 6 Million records. Even issuing systemctl stop elasticsearch.service took forever. Is this what you were looking for, if not let me know what you are looking for and I will reply ASAP |
|
there are lots of open questions, do you have some logs telling why the shards where unassigned? did you just upgrade? Why do you force them to allocate? did you run into any disk space issues? |
|
Yes, the disk got full and then after the issue started happening as ES Regards, On Dec 9, 2015, at 9:11 PM, Simon Willnauer notifications@github.com there are lots of open questions, do you have some logs telling why the — |
clintongormley
added
feedback_needed
:Distributed/Recovery
|
Here's the log around that time. |
|
OK, so here the disk is full. What happened in the logs after you cleared out space on the disk? |
|
Here's the log when I tried to start the ES after shutting it down: |
|
with your last log I can't see exceptions that indicate that recovery failed. Does the cluster come back up or do you get stuck in recoveries? The |
|
Here's the log where there's unhandled exception. Also, the cluster stuck in recovery for more than 4 hours, before I gave up and started removing indices giving problem. Basically, shards were in UNASSIGNED state and some were in INITIALIZING state. I can upload the whole log file which about 42MB if that would help (for entire day). |
please
I can't see why this is happening. so the logs would be awesome |
|
dc20151208 - was when the disk was not full but about to get full |
|
the interesting exceptions are here: I still need to investigate what's going on but can you tell me what system you are running this on? Is this a local machine or a cloud machine? I am also curious what filesystem you are using? |
|
Its a standalone node. |
|
alright I think I found the issue here @kpcool your logfiles brought the conclusion thanks you very much. This is actually a serious issue with our transaction log which basically corrupts itself when you hit a disk-full exception. I will keep you posted on this issue. Thanks for baring with me and helping to figure this out. |
s1monw
added
>bug
blocker
v5.0.0-alpha1
v2.2.0
v2.0.2
v2.1.1
:Translog
and removed
feedback_needed
labels
|
What happens here is that when we hit a disk full expection while we are flushing the transaction log we might be able to write a portion of the data but we will try to flush the entire data block over and over again. Yet, in the most of the scenarios the disk-full happens during a merge and that merge will fail and release disk-space. Once that is done we might be able to flush the translog again but we already wrote big chunks of data to disk which are now 1. corrupted and 2. treated as non-existing since our internal offsets haven't advanced. |
s1monw
added a commit
to s1monw/elasticsearch
that referenced
this issue
Dec 14, 2015
Today we are super lenient (how could I missed that for f**k sake) with failing / closing the translog writer when we hit an exception. It's actually worse, we allow to further write to it and don't care what has been already written to disk and what hasn't. We keep the buffer in memory and try to write it again on the next operation. When we hit a disk-full expcetion due to for instance a big merge we are likely adding document to the translog but fail to write them to disk. Once the merge failed and freed up it's diskspace (note this is a small window when concurrently indexing and failing the shard due to out of space exceptions) we will allow in-flight operations to add to the translog and then once we fail the shard fsync it. These operations are written to disk and fsynced which is fine but the previous buffer flush might have written some bytes to disk which are not corrupting the translog. That wouldn't be an issue if we prevented the fsync. Closes elastic#15333
|
Encountered this issue on one of the older clusters after disk full issue. Is there any way to recover the index? Losing something from the translog is not a big issue for me. Here is the expection while starting the node: These are the contents of the index translog directory: @s1monw Is there any theoretical chance to fix this? Maybe removing part of the translog a persuading elasticsearch it is complete? |
|
FYI I saw this same error in an Elasticsearch 2.3.3 environment that ran out of disk space. I was surprised I had to restart Elasticsearch in order to recover. |
clintongormley
added
:Distributed/Distributed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Today, the disk got full and ElasticSearch is not able to go back again. Isn't there a built-in system that prevents such failures. I agree that we should be monitoring the hard space and not let this happen in first place, but some times things happen.
My setup is a single node at present. Using ES 2.1.0, which was supposed to have this fix.
I don't see a clear way to recover the node. A post at https://t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode.html seemed to help, but still few indices got corrupted and I have no way to recovering them.
At the end, I ended up deleted the indices, but that's not the way it should be. Such things must be taken care of ultimately. But this is clearly a bug with ES 2.1.0
The text was updated successfully, but these errors were encountered: