Resiliency: Cancelling a recovery may leave temporary files behind #7893
Labels
Comments
clintongormley
changed the title
bleskes
added a commit
to bleskes/elasticsearch
that referenced
this issue
Oct 7, 2014
At the moment, we leave around temporary files if a peer (replica) recovery is canceled. Those files will normally be cleaned up once the shard is started else but in case of errors this can lead to trouble. If recovery are started and canceled often, we may cause nodes to run out of disk space. Closes elastic#7893
bleskes
added a commit
to bleskes/elasticsearch
that referenced
this issue
Oct 14, 2014
At the moment, we leave around temporary files if a peer (replica) recovery is canceled. Those files will normally be cleaned up once the shard is started else but in case of errors this can lead to trouble. If recovery are started and canceled often, we may cause nodes to run out of disk space. Closes elastic#7893
|
fixed with #8092 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We currently cancel recoveries when the shard is no longer assigned to the target node, or the primary shard (source of copying) is moved to another node (and there are more scenarios). That cancel logic doesn't clean up any temporary files created during the recovery.
Normally that's not a problem as the files will be cleaned up once the shard is safely recovered somewhere else (or locally). However, if one runs into continuous failure cycles we can fill up disk space, causing bigger problems like corrupting other shards on the node.
The text was updated successfully, but these errors were encountered: