Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39350

romseygeek · 2019-02-25T10:03:17Z

Contains the following:

LUCENE-8635: Move terms dictionary off-heap for non-primary-key fields in MMapDirectory
LUCENE-8292: TermsEnum is fully abstract
LUCENE-8679: Return WITHIN in EdgeTree#relateTriangle only when polygon and triangle share one edge
LUCENE-8676: Nori tokenizer deals correctly with large buffers
LUCENE-8697: GraphTokenStreamFiniteStrings better handles side paths with gaps
LUCENE-8664: Add equals and hashCode to TotalHits
LUCENE-8660: TopDocsCollector returns accurate hit counts if the total equals the threshold
LUCENE-8654: Polygon2D#relateTriangle fix for when the polygon is inside the triangle
LUCENE-8645: Intervals#fixField can merge intervals from different fields
LUCENE-8585: Create jump-tables for DocValues at index time

elasticmachine · 2019-02-25T10:03:19Z

Pinging @elastic/es-search

elasticmachine · 2019-02-25T10:05:29Z

Pinging @elastic/es-distributed

jpountz

Now that FilterTermsEnum#seekExact(BytesRef, TermState) delegates to the wrapper instance rather than to FilterTermsEnum.seekExact(BytesRef), I think it'd be safer if we checked whether the term is accepted in FieldNamesTermsEnum#seekExact(BytesRef, TermState).

romseygeek · 2019-02-25T10:54:11Z

Looks like there might be a BWC issue in the Lucene80 Codec for norms:

org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene70NormsMetadata vs expected codec=Lucene80NormsMetadata (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/var/lib/jenkins/workspace/elastic+elasticsearch+pull-request-bwc/qa/full-cluster-restart/build/cluster/v7.0.0#oldClusterTestCluster node1/elasticsearch-7.0.0-SNAPSHOT/data/nodes/0/indices/zYyufqjjSJuLRCN2RdQP-Q/2/index/_b.cfs") [slice=_b.nvm]))

romseygeek · 2019-02-25T11:38:29Z

OK, BWC tests fail here because it's targetting 7.0.0, which currently is on a previous lucene snapshot without jump tables, so the indexes are not readable across both versions. Should I disable BWC tests while this PR is merged and backported?

…a non-filtered reader

romseygeek · 2019-02-25T11:56:12Z

I think it'd be safer if we checked whether the term is accepted in FieldNamesTermsEnum#seekExact(BytesRef, TermState)

I updated to throw an IllegalStateException if the term isn't accepted, as this would mean that the TermState was from the underlying reader without being wrapped in the filter.

romseygeek · 2019-02-25T14:02:13Z

Failure in ci/2 is due to #32299

@elasticmachine run elasticsearch-ci/2

jpountz

Thanks for the ISE, LGTM.

romseygeek · 2019-02-25T14:22:38Z

Packaging tests were still running with BWC enabled

@elasticmachine test elasticsearch-ci/packaging-sample

romseygeek · 2019-02-25T14:23:13Z

@elasticmachine run elasticsearch-ci/packaging-sample

jasontedor

I left a question.

jasontedor · 2019-02-25T15:13:20Z

build.gradle

-boolean bwc_tests_enabled = true
-final String bwc_tests_disabled_issue = "" /* place a PR link here when committing bwc changes */
+boolean bwc_tests_enabled = false
+final String bwc_tests_disabled_issue = "https://github.com/elastic/elasticsearch/pull/39350" /* place a PR link here when committing bwc changes */


What is the reason for disabling all BWC tests with this PR?

BWC tests are currently being run against the 7.0.0 branch, which is using a previous lucene 8.0.0 snapshot. There has been a change in the Codec format between these two snapshots, which means that an elasticsearch running with this change cannot read lucene indexes built with the previous snapshot; upgrade tests between the current 7.0.0 branch head and this PR fail because the indexes cannot be opened.

If there's a finer-grained BWC suspension I should be doing instead, please let me know!

This also means that people using 7.0-alpha and 7.0-beta won't be able to do a rolling upgrade to 7.0-GA, because they're using unreleased lucene snapshots that do not contain this Codec change.

This relates to #38919. The issue here being that that change is still under discussion. Let me see if we can accelerate it.

This also means that people using 7.0-alpha and 7.0-beta won't be able to do a rolling upgrade to 7.0-GA, because they're using unreleased lucene snapshots that do not contain this Codec change.

That's okay, we do not support that to begin with. 😇

I merged and back-ported #38919 . The discussion is still ongoing, but we should not hold anything up for it.

romseygeek · 2019-02-25T17:08:14Z

It looks as though I need to separately disable BWC on the packaging tests?

Backport of elastic#39350

dadoonet · 2019-02-27T11:26:27Z

Side note, when we release a version, it would be helpful to think about updating this page? Not sure if this should be done before the release though as for example 7.0 branch has been yet released with 7.0.0-beta1 and users must use the previous Snapshot Lucene version when using HL Client 7.0.0-beta1 and not the one that this PR brings.

Backport of #39350 Contains the following: * LUCENE-8635: Move terms dictionary off-heap for non-primary-key fields in `MMapDirectory` * LUCENE-8292: `TermsEnum` is fully abstract * LUCENE-8679: Return WITHIN in `EdgeTree#relateTriangle` only when polygon and triangle share one edge * LUCENE-8676: Nori tokenizer deals correctly with large buffers * LUCENE-8697: `GraphTokenStreamFiniteStrings` better handles side paths with gaps * LUCENE-8664: Add `equals` and `hashCode` to `TotalHits` * LUCENE-8660: `TopDocsCollector` returns accurate hit counts if the total equals the threshold * LUCENE-8654: `Polygon2D#relateTriangle` fix for when the polygon is inside the triangle * LUCENE-8645: `Intervals#fixField` can merge intervals from different fields * LUCENE-8585: Create jump-tables for DocValues at index time

* master: Upgrade to lucene 8.0.0-snapshot-ff9509a8df (elastic#39350)

Follow-up to #39350

Upgrade to lucene 8.0.0-snapshot-ff9509a8df

22049bc

romseygeek added :Search/Search Search-related issues that do not fall into other categories >upgrade v7.0.0 v8.0.0 v7.2.0 labels Feb 25, 2019

romseygeek self-assigned this Feb 25, 2019

romseygeek requested review from jpountz and s1monw February 25, 2019 10:03

romseygeek added the :Distributed/Engine Anything around managing Lucene and the Translog in an open shard. label Feb 25, 2019

romseygeek removed the :Search/Search Search-related issues that do not fall into other categories label Feb 25, 2019

jpountz requested changes Feb 25, 2019

View reviewed changes

Refuse seekExact() with TermState if term state has been pulled from …

dd222df

…a non-filtered reader

Disabled BWC tests

f795ceb

jpountz approved these changes Feb 25, 2019

View reviewed changes

jasontedor reviewed Feb 25, 2019

View reviewed changes

Merge remote-tracking branch 'origin/master' into upgrade-lucene

2096cde

s1monw approved these changes Feb 26, 2019

View reviewed changes

Merge remote-tracking branch 'origin/master' into upgrade-lucene

9e35305

romseygeek added a commit to romseygeek/elasticsearch that referenced this pull request Feb 27, 2019

Upgrade to lucene 8.0.0-snapshot-ff9509a8df

540e7b0

Backport of elastic#39350

romseygeek mentioned this pull request Feb 27, 2019

Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39444

Merged

romseygeek added a commit to romseygeek/elasticsearch that referenced this pull request Feb 27, 2019

Upgrade to lucene 8.0.0-snapshot-ff9509a8df

202eb81

Backport of elastic#39350

romseygeek mentioned this pull request Feb 27, 2019

Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39445

Merged

romseygeek merged commit 68c57ad into elastic:master Feb 27, 2019

romseygeek deleted the upgrade-lucene branch February 27, 2019 14:38

romseygeek mentioned this pull request Feb 27, 2019

Re-enable backwards compatibility #39459

Merged

jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Feb 27, 2019

Merge branch 'master' into https-lucene-snapshots

2559239

* master: Upgrade to lucene 8.0.0-snapshot-ff9509a8df (elastic#39350)

romseygeek added a commit that referenced this pull request Feb 27, 2019

Re-enable backwards compatibility (#39459)

4ba83cd

Follow-up to #39350

michaelbaamonde added v7.0.0-rc1 and removed v7.0.0 labels Mar 25, 2019

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39350

Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39350

romseygeek commented Feb 25, 2019 •

edited

Loading

elasticmachine commented Feb 25, 2019

elasticmachine commented Feb 25, 2019

jpountz left a comment

romseygeek commented Feb 25, 2019

romseygeek commented Feb 25, 2019

romseygeek commented Feb 25, 2019

romseygeek commented Feb 25, 2019

jpountz left a comment

romseygeek commented Feb 25, 2019

romseygeek commented Feb 25, 2019

jasontedor left a comment •

edited

Loading

jasontedor Feb 25, 2019

romseygeek Feb 25, 2019

romseygeek Feb 25, 2019

jasontedor Feb 25, 2019

alpar-t Feb 26, 2019

romseygeek commented Feb 25, 2019

dadoonet commented Feb 27, 2019

Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39350

Upgrade to lucene 8.0.0-snapshot-ff9509a8df #39350

Conversation

romseygeek commented Feb 25, 2019 • edited Loading

elasticmachine commented Feb 25, 2019

elasticmachine commented Feb 25, 2019

jpountz left a comment

Choose a reason for hiding this comment

romseygeek commented Feb 25, 2019

romseygeek commented Feb 25, 2019

romseygeek commented Feb 25, 2019

romseygeek commented Feb 25, 2019

jpountz left a comment

Choose a reason for hiding this comment

romseygeek commented Feb 25, 2019

romseygeek commented Feb 25, 2019

jasontedor left a comment • edited Loading

Choose a reason for hiding this comment

jasontedor Feb 25, 2019

Choose a reason for hiding this comment

romseygeek Feb 25, 2019

Choose a reason for hiding this comment

romseygeek Feb 25, 2019

Choose a reason for hiding this comment

jasontedor Feb 25, 2019

Choose a reason for hiding this comment

alpar-t Feb 26, 2019

Choose a reason for hiding this comment

romseygeek commented Feb 25, 2019

dadoonet commented Feb 27, 2019

romseygeek commented Feb 25, 2019 •

edited

Loading

jasontedor left a comment •

edited

Loading