Mapping: Remove pulsing/bloom_pulsing postings format #7566

Closed
mikemccand opened this Issue Sep 3, 2014 · 3 comments

Projects

None yet

4 participants

@mikemccand
Contributor

In #7238 it looked like index corruption (checksum errors) but in fact it was simply that the user selected bloom_pulsing postings format, which we don't support yet still allow.

We recently removed documentation showing these postings format as a choice, but it's still really dangerous we allow this option at all since it creates unusable indices in ES when we migrate shards and try to check integrity. Before 1.3, ES didn't check Lucene checksums, so these postings formats worked fine, but with 1.3 any index using pulsing will fail.

The pulsing optimization has already been folded into the default postings format for quite a while now.

I think we should remove them; we are already removing pulsing from Lucene (https://issues.apache.org/jira/browse/LUCENE-5915)

@rmuir
Contributor
rmuir commented Sep 3, 2014

Should we go further and disable (for now) any custom formats that don't have backwards compatibility support from lucene? These can change across releases in such a way that looks like corruption.

We are currently trying to figure out a way in Lucene to safely provide options to the user AND backwards compatibility, but this is not going to happen overnight.

@mikemccand
Contributor

Should we go further and disable (for now) any custom formats that don't have backwards compatibility support from lucene? These can change across releases in such a way that looks like corruption.

+1, I'll do that.

@s1monw
Contributor
s1monw commented Sep 3, 2014

++ to removing all the non-checksumming formats!

@mikemccand mikemccand added a commit that closed this issue Sep 8, 2014
@mikemccand mikemccand Core: remove built-in support for Lucene's experimental codecs
Lucene's experimental codecs (from the codecs module) do not provide
backwards compatibility and are free to change from release to
release.  When they do change, they typically cannot in general read
older indices and the resulting exceptions look like index corruption.
So, we are removing built-in support for them to prevent applications
from choosing one and then seeing strange exceptions on upgrade.

Closes #7566

Closes #7604
130fdef
@mikemccand mikemccand closed this in 130fdef Sep 8, 2014
@mikemccand mikemccand added a commit that referenced this issue Sep 8, 2014
@mikemccand mikemccand Core: remove built-in support for Lucene's experimental codecs
Lucene's experimental codecs (from the codecs module) do not provide
backwards compatibility and are free to change from release to
release.  When they do change, they typically cannot in general read
older indices and the resulting exceptions look like index corruption.
So, we are removing built-in support for them to prevent applications
from choosing one and then seeing strange exceptions on upgrade.

Closes #7566

Closes #7604
f8096a8
@clintongormley clintongormley changed the title from Remove pulsing/bloom_pulsing postings format to Mapping: Remove pulsing/bloom_pulsing postings format Sep 8, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment