Translog entry checksums #6554

Closed
dakrone opened this Issue Jun 18, 2014 · 3 comments

Projects

None yet

3 participants

@dakrone
Member
dakrone commented Jun 18, 2014

It's possible that a transaction log can be corrupted on disk, causing it to throw exceptions like:

Exception in thread "main" org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [108]
    at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
    at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:506)
    at org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
    at org.writequit.Main.main(Main.java:22)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

It would be nice if we had checksums for each entry in the translog, to determine corruption in a more-friendly manner than throwing an exception with "No version type match".

@dakrone dakrone added the enhancement label Jun 18, 2014
@dakrone
Member
dakrone commented Jun 18, 2014

To clarify this, we should add checksums first, and then optionally discuss markers and entry skipping.

@areek areek assigned rmuir and unassigned rmuir Jun 23, 2014
@dakrone dakrone self-assigned this Jul 25, 2014
@dakrone dakrone added a commit to dakrone/elasticsearch that referenced this issue Aug 27, 2014
@dakrone dakrone Add translog checksums
Switches TranslogStreams to check a header in the file to determine the
translog format, delegating to the version-specific stream.

Version 1 of the translog format writes a header using Lucene's
CodecUtil at the beginning of the file and appends a checksum for each
translog operation written.

Also refactors much of the translog operations, such as merging
.hasNext() and .next() in FsChannelSnapshot

Relates to #6554
eaf3921
@dakrone dakrone added a commit that referenced this issue Aug 27, 2014
@dakrone dakrone Add translog checksums
Switches TranslogStreams to check a header in the file to determine the
translog format, delegating to the version-specific stream.

Version 1 of the translog format writes a header using Lucene's
CodecUtil at the beginning of the file and appends a checksum for each
translog operation written.

Also refactors much of the translog operations, such as merging
.hasNext() and .next() in FsChannelSnapshot

Relates to #6554
85ba3be
@dakrone dakrone added a commit that referenced this issue Sep 8, 2014
@dakrone @areek dakrone + areek Add translog checksums
Switches TranslogStreams to check a header in the file to determine the
translog format, delegating to the version-specific stream.

Version 1 of the translog format writes a header using Lucene's
CodecUtil at the beginning of the file and appends a checksum for each
translog operation written.

Also refactors much of the translog operations, such as merging
.hasNext() and .next() in FsChannelSnapshot

Relates to #6554
d344419
@s1monw
Contributor
s1monw commented Sep 10, 2014

ping @dakrone what is the status here?

@dakrone dakrone changed the title from Translog entry checksums and entry markers to Translog entry checksums Sep 10, 2014
@dakrone
Member
dakrone commented Sep 10, 2014

opened #7668 for the marking feature and resolving this since the translog checksums work has been merged to master and 1.x.

@dakrone dakrone closed this Sep 10, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment