Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add translog checksums #7232

Merged
merged 1 commit into from Aug 27, 2014
Merged

Conversation

dakrone
Copy link
Member

@dakrone dakrone commented Aug 12, 2014

Switches TranslogStreams to check a header in the file to determine the
translog format, delegating to the version-specific stream.

Version 1 of the translog format writes a 0xffff_ffff_0000_0001 header
at the beginning of the file and appends a checksum for each translog
operation written.

Also refactors much of the translog operations, such as merging
.hasNext() and .next() in FsChannelSnapshot

Relates to #6554

@@ -269,7 +268,9 @@ public void recover(boolean indexShouldExists, RecoveryState recoveryState) thro
throw new IndexShardGatewayRecoveryException(shardId, "failed to recover shard", e);
} finally {
try {
fs.close();
if (stream != null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe use IOUtils.close() handles null values

@s1monw
Copy link
Contributor

s1monw commented Aug 20, 2014

I left a bunch of comments. I personally think we should add the checksum calculation on the TranslogStream level somehow instead of doing it on each operation. we should have an StreamOutput impl that calculates the checksum as well as a StreamInput that makes it entirely transparent. I am also worried about the potential collision of the header so I think we should make use of the index version to detect if we can use the new version of the translog

* A CRC32 checksum class that has helpers for adding strings and numeric types
*/
public class ChecksumHelper {
private final Checksum checksum = new CRC32();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

each invocation of CRC32 is pretty slow: if we do it byte-by-byte its a native call, and we cant use vectorized bulk processing. can you try:

import org.apache.lucene.store.BufferedChecksum;
...
private final Checksum checksum = new BufferedChecksum(new CRC32());

This should improve performance significantly.

@s1monw s1monw removed the review label Aug 21, 2014

@Override
public int writeHeader(FileChannel channel) throws IOException {
OutputStreamDataOutput out = new OutputStreamDataOutput(Channels.newOutputStream(channel));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we put a comment here that we don't close on purpose?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes good idea, always good to make the intent known, I'll do that.

* Similar to Lucene's BufferedChecksumIndexOutput, however this wraps a
* {@link StreamOutput} so anything written will update the checksum
*/
public class BufferedChecksumStreamOutput extends StreamOutput {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it make sense to make these classes final?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it definitely does, I'll change that.

* the transient or current translog does not match, returns null
*/
private FsTranslogFile translogForLocation(Location location) {
if (this.trans != null && trans.id() == location.translogId) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can remove the this qualifier for consistency?

@s1monw
Copy link
Contributor

s1monw commented Aug 25, 2014

I left some more comments, I think it's close though... I would really really like to see a test that does corrupt a translog on a node and then checks that we handle it correctly... I am not even sure what the correct behavior is but IMO we should fail the shard?

@s1monw s1monw removed the review label Aug 25, 2014
}

@Override
public int read() throws IOException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the contract of this method? Usually (e.g. java InputStream) its that it returns an integer value between 0 and 255. So I think this is missing & 0xFF ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, added this, thanks!

@dakrone dakrone added the review label Aug 26, 2014
@dakrone
Copy link
Member Author

dakrone commented Aug 27, 2014

@s1monw added an integration test for this.

@s1monw
Copy link
Contributor

s1monw commented Aug 27, 2014

@dakrone this test looks awesome. Can we maybe add that it tells you if it corrupted a replica and if so we make sure that the replica is allocated on another node? if you wanna do that afterwards I am ok with pushing this as it is...

@s1monw s1monw removed the review label Aug 27, 2014
@dakrone dakrone force-pushed the feature/translog-checksums branch 3 times, most recently from 0ca533d to eaf3921 Compare August 27, 2014 13:17
Switches TranslogStreams to check a header in the file to determine the
translog format, delegating to the version-specific stream.

Version 1 of the translog format writes a header using Lucene's
CodecUtil at the beginning of the file and appends a checksum for each
translog operation written.

Also refactors much of the translog operations, such as merging
.hasNext() and .next() in FsChannelSnapshot

Relates to elastic#6554
@dakrone dakrone merged commit eaf3921 into elastic:master Aug 27, 2014
@clintongormley clintongormley changed the title Add translog checksums Resiliency: Add translog checksums Sep 8, 2014
@dakrone dakrone deleted the feature/translog-checksums branch September 9, 2014 13:52
@clintongormley clintongormley changed the title Resiliency: Add translog checksums Add translog checksums Jun 7, 2015
@clintongormley clintongormley added :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. :Distributed/Engine Anything around managing Lucene and the Translog in an open shard. and removed :Translog :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. labels Feb 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/Engine Anything around managing Lucene and the Translog in an open shard. >enhancement resiliency v1.4.0.Beta1 v2.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants