Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Checksum state files written to disk #7586

Closed
dakrone opened this issue Sep 4, 2014 · 0 comments · Fixed by #8010
Closed

Checksum state files written to disk #7586

dakrone opened this issue Sep 4, 2014 · 0 comments · Fixed by #8010
Assignees
@s1monw
Labels
>enhancement resiliency v1.5.0 v2.0.0-beta1

Comments

@dakrone
Copy link
Member

dakrone commented Sep 4, 2014

We write a number of state files to disk, including persistent cluster settings, the state of indices and so forth. In order to detect disk corruption in these files we should also write the checksum of these files to disk. On read we can verify the integrity of these files if a checksum file is present.
@dakrone dakrone self-assigned this Sep 4, 2014
@s1monw s1monw assigned s1monw and unassigned dakrone Oct 6, 2014
@s1monw s1monw mentioned this issue Oct 7, 2014
Merged
s1monw added a commit to s1monw/elasticsearch that referenced this issue Oct 15, 2014
@s1monw
[CORE] Add cluster and index state checksums
8dddac3 
This commit adds checksumming for cluster and index states. It moves
from a plain XContent based on-disk format to a more structured binary
format including a header and footer as well as a CRC32 checksum for
each of these files. Since previous versions didn't write any format
identifier etc. this commit adds a file extension `.st` for states
that have header/footer and checksum.
This commit also moves over to write temporary files and moves them into
place with an atomic move operation. This commit also serializes and
deserializes the states without materilazing them into opaque memory.

Closes elastic#7586
s1monw added a commit that referenced this issue Oct 15, 2014
@s1monw
[CORE] Add cluster and index state checksums
c399509 
This commit adds checksumming for cluster and index states. It moves
from a plain XContent based on-disk format to a more structured binary
format including a header and footer as well as a CRC32 checksum for
each of these files. Since previous versions didn't write any format
identifier etc. this commit adds a file extension `.st` for states
that have header/footer and checksum.
This commit also moves over to write temporary files and moves them into
place with an atomic move operation. This commit also serializes and
deserializes the states without materilazing them into opaque memory.

Closes #7586
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@s1monw s1monw

Labels
>enhancement resiliency v1.5.0 v2.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add cluster and index state checksums s1monw/elasticsearch
2 participants
@dakrone @s1monw