Harden recovery for old segments #8399
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a lucene 4.8+ file is transferred, Store returns a VerifyingIndexOutput that verifies both the CRC32 integrity and the length of the file. However, for older files, problems can make it to the lucene level. This is not great since older lucene files aren't especially strong as far as detecting issues here. For example, if a network transfer is closed on the remote side, we might write a truncated file... which old lucene formats may or may not detect. The idea here is to verify old files with their legacy Adler32 checksum, plus expected length. If they don't have an Adler32 (segments_N, jurassic elasticsearch?, its optional as far as the protocol goes), then at least check the length. We could improve it for segments_N, its had an embedded CRC32 forever in lucene, but this gets trickier. Long term, we should also try to also improve tests around here, especially backwards compat testing, we should test that detected corruptions are handled properly.
|
LGTM! |
|
I didn't look very close but on a high level this looks very good to me - not sure why we haven't done this before to be honest... |
| /** | ||
| * verifies length for index files before lucene 4.8 | ||
| */ | ||
| static class LengthVerifyingIndexOutput extends VerifyingIndexOutput { |
There was a problem hiding this comment.
seems like a FilterIndexOutput is worth putting into lucene?
There was a problem hiding this comment.
Maybe, although Lucene doesn't decorate at this level. Instead, it works at OutputStream level, using JDK classes.
For example the default chain is essentially:
FilterOutputStream(
BufferedOutputStream(
CheckedOutputStream(
Files.newOutputStream(...))))
|
left some rather cosmetic comments, LGTM though - honestly I almost classify this as a bugfix so I think we should push it to |
|
Thanks for looking @s1monw . Can you look at my comments? I am concerned that those comments are not cosmetic but are too risky for this task. |
|
I tried a way that might make us both happy. I made the irrelevant forwarded methods 'final' and so on. |
|
thanks @rmuir LGTM |
|
LGTM |
rmuir
added a commit
that referenced
this pull request
Nov 9, 2014
When a lucene 4.8+ file is transferred, Store returns a VerifyingIndexOutput that verifies both the CRC32 integrity and the length of the file. However, for older files, problems can make it to the lucene level. This is not great since older lucene files aren't especially strong as far as detecting issues here. For example, if a network transfer is closed on the remote side, we might write a truncated file... which old lucene formats may or may not detect. The idea here is to verify old files with their legacy Adler32 checksum, plus expected length. If they don't have an Adler32 (segments_N, jurassic elasticsearch?, its optional as far as the protocol goes), then at least check the length. We could improve it for segments_N, its had an embedded CRC32 forever in lucene, but this gets trickier. Long term, we should also try to also improve tests around here, especially backwards compat testing, we should test that detected corruptions are handled properly. Closes #8399
rmuir
added a commit
that referenced
this pull request
Nov 9, 2014
When a lucene 4.8+ file is transferred, Store returns a VerifyingIndexOutput that verifies both the CRC32 integrity and the length of the file. However, for older files, problems can make it to the lucene level. This is not great since older lucene files aren't especially strong as far as detecting issues here. For example, if a network transfer is closed on the remote side, we might write a truncated file... which old lucene formats may or may not detect. The idea here is to verify old files with their legacy Adler32 checksum, plus expected length. If they don't have an Adler32 (segments_N, jurassic elasticsearch?, its optional as far as the protocol goes), then at least check the length. We could improve it for segments_N, its had an embedded CRC32 forever in lucene, but this gets trickier. Long term, we should also try to also improve tests around here, especially backwards compat testing, we should test that detected corruptions are handled properly. Closes #8399 Conflicts: src/main/java/org/elasticsearch/index/store/Store.java
rmuir
added a commit
that referenced
this pull request
Nov 9, 2014
When a lucene 4.8+ file is transferred, Store returns a VerifyingIndexOutput that verifies both the CRC32 integrity and the length of the file. However, for older files, problems can make it to the lucene level. This is not great since older lucene files aren't especially strong as far as detecting issues here. For example, if a network transfer is closed on the remote side, we might write a truncated file... which old lucene formats may or may not detect. The idea here is to verify old files with their legacy Adler32 checksum, plus expected length. If they don't have an Adler32 (segments_N, jurassic elasticsearch?, its optional as far as the protocol goes), then at least check the length. We could improve it for segments_N, its had an embedded CRC32 forever in lucene, but this gets trickier. Long term, we should also try to also improve tests around here, especially backwards compat testing, we should test that detected corruptions are handled properly. Closes #8399 Conflicts: src/main/java/org/elasticsearch/index/store/Store.java
rmuir
added a commit
that referenced
this pull request
Nov 9, 2014
When a lucene 4.8+ file is transferred, Store returns a VerifyingIndexOutput that verifies both the CRC32 integrity and the length of the file. However, for older files, problems can make it to the lucene level. This is not great since older lucene files aren't especially strong as far as detecting issues here. For example, if a network transfer is closed on the remote side, we might write a truncated file... which old lucene formats may or may not detect. The idea here is to verify old files with their legacy Adler32 checksum, plus expected length. If they don't have an Adler32 (segments_N, jurassic elasticsearch?, its optional as far as the protocol goes), then at least check the length. We could improve it for segments_N, its had an embedded CRC32 forever in lucene, but this gets trickier. Long term, we should also try to also improve tests around here, especially backwards compat testing, we should test that detected corruptions are handled properly. Closes #8399 Conflicts: src/main/java/org/elasticsearch/index/store/Store.java src/test/java/org/elasticsearch/index/store/StoreTest.java
clintongormley
added
:Distributed/Recovery
clintongormley
changed the title
mute
pushed a commit
to mute/elasticsearch
that referenced
this pull request
Jul 29, 2015
When a lucene 4.8+ file is transferred, Store returns a VerifyingIndexOutput that verifies both the CRC32 integrity and the length of the file. However, for older files, problems can make it to the lucene level. This is not great since older lucene files aren't especially strong as far as detecting issues here. For example, if a network transfer is closed on the remote side, we might write a truncated file... which old lucene formats may or may not detect. The idea here is to verify old files with their legacy Adler32 checksum, plus expected length. If they don't have an Adler32 (segments_N, jurassic elasticsearch?, its optional as far as the protocol goes), then at least check the length. We could improve it for segments_N, its had an embedded CRC32 forever in lucene, but this gets trickier. Long term, we should also try to also improve tests around here, especially backwards compat testing, we should test that detected corruptions are handled properly. Closes elastic#8399 Conflicts: src/main/java/org/elasticsearch/index/store/Store.java
mute
pushed a commit
to mute/elasticsearch
that referenced
this pull request
Jul 29, 2015
When a lucene 4.8+ file is transferred, Store returns a VerifyingIndexOutput that verifies both the CRC32 integrity and the length of the file. However, for older files, problems can make it to the lucene level. This is not great since older lucene files aren't especially strong as far as detecting issues here. For example, if a network transfer is closed on the remote side, we might write a truncated file... which old lucene formats may or may not detect. The idea here is to verify old files with their legacy Adler32 checksum, plus expected length. If they don't have an Adler32 (segments_N, jurassic elasticsearch?, its optional as far as the protocol goes), then at least check the length. We could improve it for segments_N, its had an embedded CRC32 forever in lucene, but this gets trickier. Long term, we should also try to also improve tests around here, especially backwards compat testing, we should test that detected corruptions are handled properly. Closes elastic#8399 Conflicts: src/main/java/org/elasticsearch/index/store/Store.java
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a lucene 4.8+ file is transferred, Store returns a VerifyingIndexOutput
that verifies both the CRC32 integrity and the length of the file.
However, for older files, problems can make it to the lucene level. This is not great
since older lucene files aren't especially strong as far as detecting issues here.
For example, if a network transfer is closed on the remote side, we might write a
truncated or corrupted file... which old lucene formats may or may not detect.
The idea here is to verify old files with their legacy Adler32 checksum, plus expected
length. If they don't have an Adler32 (segments_N, jurassic elasticsearch?, its optional
as far as the protocol goes), then at least check the length.
We could improve it for segments_N, its had an embedded CRC32 forever in lucene, but this
gets trickier. Long term, we should also try to also improve tests around here, especially
backwards compat testing, we should test that detected corruptions are handled properly.
I added unit tests for the two cases covered here. PR is against 1.x since thats what i'm working with,
but I can forward-port to master once we figure this out.