Verify checksums on merge #7360

Closed
wants to merge 7 commits into
from

Projects

None yet

4 participants

@rmuir
Contributor
rmuir commented Aug 21, 2014

This just exposes the current lucene option checkIntegrityAtMerge from LiveIndexWriterConfig. When enabled, all parts of the index are verified before merging.

@rjernst rjernst and 2 others commented on an outdated diff Aug 21, 2014
docs/reference/index-modules.asciidoc
@@ -35,6 +35,11 @@ otherwise it is written in non-compound format.
Should a new segment (create by indexing, not by merging) be written
in compound format or non-compound format? Defaults to `true`.
This is a dynamic setting.
+
+[[index-checksum-on-merge]]`index.checksum_on_merge`::
+
+ Should merging first validate checksums on segments being merged?
+ Defaults to `false`. This is a dynamic setting. coming[1.4.0]
@rjernst
rjernst Aug 21, 2014 Member

Should this be "Added [1.4.0]"? No one will see this in official docs until then right?

@rmuir
rmuir Aug 21, 2014 Contributor

afaik coming is automatically changed to added during the release process.

@rjernst
rjernst Aug 21, 2014 Member

Oh wow, didn't know that...

@s1monw
s1monw Aug 21, 2014 Contributor

yeah at this point we build our docs from the 1.x branch so your chance as immediate effect. It will be moved to added once we release it.

@rjernst rjernst commented on an outdated diff Aug 21, 2014
docs/reference/indices/update-settings.asciidoc
@@ -155,6 +155,12 @@ added[1.0.0.RC1]
`index.warmer.enabled`::
See <<indices-warmers>>. Defaults to `true`.
+
+coming[1.4.0]
@rjernst
rjernst Aug 21, 2014 Member

Ditto. "added"?

@rjernst
Member
rjernst commented Aug 21, 2014

This looks good. Just a minor comment on the docs.

@s1monw s1monw commented on the diff Aug 21, 2014
...search/index/engine/internal/InternalEngineTests.java
@@ -113,6 +113,7 @@ public void setUp() throws Exception {
super.setUp();
defaultSettings = ImmutableSettings.builder()
.put(InternalEngine.INDEX_COMPOUND_ON_FLUSH, getRandom().nextBoolean())
+ .put(InternalEngine.INDEX_CHECKSUM_ON_MERGE, getRandom().nextBoolean())
@s1monw
s1monw Aug 21, 2014 Contributor

I think we need a test that also checks that the live change works. Since you missed to register the setting in
org.elasticsearch.index.settings.IndexDynamicSettingsModule such a test should fail.

@rmuir
rmuir Aug 21, 2014 Contributor

Yes, I really wanted this too. I was beaten back by the fact that tests for any of the other IndexWriter options are integration tests (indexing stuff and looking at number of segments etc), instead of simply being unit tests that check that IWC was changed.

@rmuir
Contributor
rmuir commented Sep 2, 2014

I updated this with a test extending simon's new base class, changing default to true.

@rjernst
Member
rjernst commented Sep 2, 2014

LGTM

@s1monw
Contributor
s1monw commented Sep 2, 2014

can you change the name of the issue to reflect the fact that we enable this feature rather than adding an option. Other than that LGTM thanks rob

@rmuir rmuir changed the title from Add index.checksum_on_merge option to Verify checksums on merge Sep 2, 2014
@rmuir rmuir added a commit that closed this pull request Sep 2, 2014
@rmuir rmuir [Engine] Verify checksums on merge
Enable lucene verification of checksums on segments before merging them.
This prevents corruption from existing segments from silently slipping into
newer merged segments.

Closes #7360
1711041
@rmuir rmuir closed this in 1711041 Sep 2, 2014
@rmuir rmuir added a commit that referenced this pull request Sep 2, 2014
@rmuir rmuir [Engine] Verify checksums on merge
Enable lucene verification of checksums on segments before merging them.
This prevents corruption from existing segments from silently slipping into
newer merged segments.

Closes #7360
9459eee
@s1monw s1monw deleted the rmuir:merge-checksum branch Sep 3, 2014
@rmuir rmuir added a commit that referenced this pull request Sep 8, 2014
@rmuir @areek rmuir + areek [Engine] Verify checksums on merge
Enable lucene verification of checksums on segments before merging them.
This prevents corruption from existing segments from silently slipping into
newer merged segments.

Closes #7360
33b77c3
@clintongormley clintongormley changed the title from Verify checksums on merge to Resiliency: Verify checksums on merge Sep 8, 2014
@clintongormley clintongormley added the :Core label Jun 7, 2015
@clintongormley clintongormley changed the title from Resiliency: Verify checksums on merge to Verify checksums on merge Jun 7, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment