Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add checksumming and versions to the Translog's Checkpoint files #19797

Merged
merged 6 commits into from
Aug 4, 2016
Merged

Add checksumming and versions to the Translog's Checkpoint files #19797

merged 6 commits into from
Aug 4, 2016

Conversation

bleskes
Copy link
Contributor

@bleskes bleskes commented Aug 4, 2016

This prepares the infrastructure to be able to extend the checkpoint file to store more information.

@bleskes
Copy link
Contributor Author

bleskes commented Aug 4, 2016

@s1monw @mikemccand can you guys take a look?

PS. I would love to get rid of the DirectOutputStreamIndexOutput I added there, but I didn't want multiple in memory buffers - it just irked me the wrong way. It will be wonderful to hear if there is a better way in the big jungle of OutputStreams and DataOutputs

mikemccand
mikemccand reviewed Aug 4, 2016
View reviewed changes
core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java
return new Checkpoint(new InputStreamDataInput(in));
try (Directory dir = new SimpleFSDirectory(path.getParent())) {
try (final IndexInput indexInput = dir.openInput(path.getFileName().toString(), IOContext.DEFAULT)) {
if (indexInput.length() == 20) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we pull 20 out to a static constant like OLD_CHECKSUM_FILE_LENGTH?

@mikemccand
Copy link
Contributor

Thanks @bleskes, I left some comments.

@bleskes
Copy link
Contributor Author

bleskes commented Aug 4, 2016 •
edited
Loading

@mikemccand @s1monw I pushed another commit addressing feedback. I switched back to vanilla OutputStreamDataOutput, which means double buffers but it seems you guys are more comfortable with that trade off (which I agree with if there is no feature coming to allow to disable the buffering in OutputStreamDataOutput)

mikemccand
mikemccand reviewed Aug 4, 2016
View reviewed changes
core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java
final ByteArrayDataOutput out = new ByteArrayDataOutput(buffer);
write(out);
Channels.writeToChannel(buffer, channel);
static Checkpoint readChecksummed(DataInput in) throws IOException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe rename to readChecksummedV1?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do.

@mikemccand
Copy link
Contributor

LGTM, I just left a minor renaming suggestion.

@s1monw
Copy link
Contributor

s1monw commented Aug 4, 2016

LGTM 2

@bleskes bleskes merged commit 7010082 into elastic:master Aug 4, 2016
@bleskes bleskes deleted the translog_versioned_checkpoint branch August 4, 2016 18:42
@clintongormley clintongormley added :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. :Distributed/Engine Anything around managing Lucene and the Translog in an open shard. and removed :Translog :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. labels Feb 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
:Distributed/Engine Anything around managing Lucene and the Translog in an open shard. >enhancement v5.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@bleskes @mikemccand @s1monw @clintongormley