-
Notifications
You must be signed in to change notification settings - Fork 24.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ElasticSearch fail on startup if found empty state file for index #27007
Comments
These state files are written atomically. That is, when we are going to write a new state file out to disk, we first write the new state file to temporary file, then we fsync the file to disk, then we atomically move the temporary file into its permanent location (the underlying filesystem must support atomic move, there is no fallback here), then we fsync the directory to disk. Seeing a zero-length state file means the file was modified by external force. Failing startup is the right behavior here, we should not leniently allow external forces to modify files on disk, this needs to be forcefully brought to your attention so that you can address the root problem, it requires manual intervention to recover from.
We simply abhor leniency of this form. If these files are modified by external force, corrupted on disk, etc. it needs to be brought to your attention. Lines in a log file are not enough. |
@jasontedor , thanks for explain! :) |
@nefelim4ag You're most welcome. |
I'm having a similar issue:
I had a node die with an OutOfMemoryError. When I tried to restart the nodes, they could not start back up and were complaining about being unable to read an index state file.
I checked the file, and unlike other index state files, the file in the log has no content
Elasticsearch will NOT startup in this state. My cluster is in a perpetual red state as I cannot allocate the unallocated primary and replica shard that has damaged state files. Here's a bit from where I tried to allocate the unassigned shards:
... If I'm reading the docs right, I'm looking at data loss if I were to use the reroute API with
|
ES 5.6.3
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-8u141-b15-1~deb9u1-b15)
OpenJDK 64-Bit Server VM (build 25.141-b15, mixed mode)
Linux ids 4.12.0-0.bpo.1-amd64 #1 SMP Debian 4.12.6-1~bpo9+1 (2017-08-27) x86_64 GNU/Linux
Some times after server crash, ES can found files like:
/ES/ES_STOR_00/nodes/0/indices/9UbB7kLtQ4mBJxPCzgHjZQ/_state/state-30.st
with zero size (empty), and fail start up.
I think ES must just ignore/drop that files and reread indexes
Thanks
The text was updated successfully, but these errors were encountered: