Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapping: Remove pulsing/bloom_pulsing postings format #7566

Closed
mikemccand opened this Issue Sep 3, 2014 · 3 comments

Comments

Projects
None yet
4 participants
@mikemccand
Copy link
Contributor

mikemccand commented Sep 3, 2014

In #7238 it looked like index corruption (checksum errors) but in fact it was simply that the user selected bloom_pulsing postings format, which we don't support yet still allow.

We recently removed documentation showing these postings format as a choice, but it's still really dangerous we allow this option at all since it creates unusable indices in ES when we migrate shards and try to check integrity. Before 1.3, ES didn't check Lucene checksums, so these postings formats worked fine, but with 1.3 any index using pulsing will fail.

The pulsing optimization has already been folded into the default postings format for quite a while now.

I think we should remove them; we are already removing pulsing from Lucene (https://issues.apache.org/jira/browse/LUCENE-5915)

@mikemccand mikemccand added bug labels Sep 3, 2014

@rmuir

This comment has been minimized.

Copy link
Contributor

rmuir commented Sep 3, 2014

Should we go further and disable (for now) any custom formats that don't have backwards compatibility support from lucene? These can change across releases in such a way that looks like corruption.

We are currently trying to figure out a way in Lucene to safely provide options to the user AND backwards compatibility, but this is not going to happen overnight.

@mikemccand

This comment has been minimized.

Copy link
Contributor Author

mikemccand commented Sep 3, 2014

Should we go further and disable (for now) any custom formats that don't have backwards compatibility support from lucene? These can change across releases in such a way that looks like corruption.

+1, I'll do that.

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Sep 3, 2014

++ to removing all the non-checksumming formats!

@mikemccand mikemccand closed this in 130fdef Sep 8, 2014

mikemccand added a commit that referenced this issue Sep 8, 2014

Core: remove built-in support for Lucene's experimental codecs
Lucene's experimental codecs (from the codecs module) do not provide
backwards compatibility and are free to change from release to
release.  When they do change, they typically cannot in general read
older indices and the resulting exceptions look like index corruption.
So, we are removing built-in support for them to prevent applications
from choosing one and then seeing strange exceptions on upgrade.

Closes #7566

Closes #7604

@clintongormley clintongormley changed the title Remove pulsing/bloom_pulsing postings format Mapping: Remove pulsing/bloom_pulsing postings format Sep 8, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.