Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify checksums on merge #7360

Closed
wants to merge 7 commits into from

Conversation

Projects
None yet
4 participants
@rmuir
Copy link
Contributor

rmuir commented Aug 21, 2014

This just exposes the current lucene option checkIntegrityAtMerge from LiveIndexWriterConfig. When enabled, all parts of the index are verified before merging.

[[index-checksum-on-merge]]`index.checksum_on_merge`::

Should merging first validate checksums on segments being merged?
Defaults to `false`. This is a dynamic setting. coming[1.4.0]

This comment has been minimized.

Copy link
@rjernst

rjernst Aug 21, 2014

Member

Should this be "Added [1.4.0]"? No one will see this in official docs until then right?

This comment has been minimized.

Copy link
@rmuir

rmuir Aug 21, 2014

Author Contributor

afaik coming is automatically changed to added during the release process.

This comment has been minimized.

Copy link
@rjernst

rjernst Aug 21, 2014

Member

Oh wow, didn't know that...

This comment has been minimized.

Copy link
@s1monw

s1monw Aug 21, 2014

Contributor

yeah at this point we build our docs from the 1.x branch so your chance as immediate effect. It will be moved to added once we release it.

@@ -155,6 +155,12 @@ added[1.0.0.RC1]

`index.warmer.enabled`::
See <<indices-warmers>>. Defaults to `true`.

coming[1.4.0]

This comment has been minimized.

Copy link
@rjernst

rjernst Aug 21, 2014

Member

Ditto. "added"?

@rjernst

This comment has been minimized.

Copy link
Member

rjernst commented Aug 21, 2014

This looks good. Just a minor comment on the docs.

@@ -113,6 +113,7 @@ public void setUp() throws Exception {
super.setUp();
defaultSettings = ImmutableSettings.builder()
.put(InternalEngine.INDEX_COMPOUND_ON_FLUSH, getRandom().nextBoolean())
.put(InternalEngine.INDEX_CHECKSUM_ON_MERGE, getRandom().nextBoolean())

This comment has been minimized.

Copy link
@s1monw

s1monw Aug 21, 2014

Contributor

I think we need a test that also checks that the live change works. Since you missed to register the setting in
org.elasticsearch.index.settings.IndexDynamicSettingsModule such a test should fail.

This comment has been minimized.

Copy link
@rmuir

rmuir Aug 21, 2014

Author Contributor

Yes, I really wanted this too. I was beaten back by the fact that tests for any of the other IndexWriter options are integration tests (indexing stuff and looking at number of segments etc), instead of simply being unit tests that check that IWC was changed.

@rmuir

This comment has been minimized.

Copy link
Contributor Author

rmuir commented Sep 2, 2014

I updated this with a test extending simon's new base class, changing default to true.

@rjernst

This comment has been minimized.

Copy link
Member

rjernst commented Sep 2, 2014

LGTM

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Sep 2, 2014

can you change the name of the issue to reflect the fact that we enable this feature rather than adding an option. Other than that LGTM thanks rob

@rmuir rmuir changed the title Add index.checksum_on_merge option Verify checksums on merge Sep 2, 2014

@rmuir rmuir closed this in 1711041 Sep 2, 2014

rmuir added a commit that referenced this pull request Sep 2, 2014

[Engine] Verify checksums on merge
Enable lucene verification of checksums on segments before merging them.
This prevents corruption from existing segments from silently slipping into
newer merged segments.

Closes #7360

@s1monw s1monw deleted the rmuir:merge-checksum branch Sep 3, 2014

rmuir added a commit that referenced this pull request Sep 8, 2014

[Engine] Verify checksums on merge
Enable lucene verification of checksums on segments before merging them.
This prevents corruption from existing segments from silently slipping into
newer merged segments.

Closes #7360

@clintongormley clintongormley changed the title Verify checksums on merge Resiliency: Verify checksums on merge Sep 8, 2014

@clintongormley clintongormley changed the title Resiliency: Verify checksums on merge Verify checksums on merge Jun 7, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.