SOLR-17097: Upgrade Solr to use Lucene 9.9.2 #2176

NazerkeBS · 2024-01-04T10:51:31Z

*https://issues.apache.org/jira/browse/SOLR-17097

Description

Upgrade Solr to use Lucene 9.9.2

Solution

Please provide a short description of the approach taken to implement your solution.

Tests

Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.

Checklist

Please review the following and check all that apply:

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.
I have added documentation for the Reference Guide

epugh · 2024-01-04T14:11:58Z

It's great that we are working to stay up to date with Lucene! Looks like two of the tests are failing?

NazerkeBS · 2024-01-04T16:37:04Z

It's great that we are working to stay up to date with Lucene! Looks like two of the tests are failing?

TestSolrQueryParser#testSyntax - re-run it a few times locally , surprisingly it passed then failed, passed, failed; somewhat flaky.

TestScoreJoinQPNoScore#testRandomJoin - took a look into it; it seems the problem relates to ScoreMode ScoreJoinQParserPlugin. Locally in my machineorg.apache.lucene.search.ScoreMode class is not found and this surprised me. Looked into jars and seeing ScoreMode.class instead of ScoreMode.

cpoerschke · 2024-01-09T17:00:11Z

solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java

+          FieldComparator<?> comparator =
+              sortField.getComparator(1, Pruning.GREATER_THAN_OR_EQUAL_TO);


ref: apache/lucene@d910990

cpoerschke · 2024-01-09T17:05:53Z

solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java

@@ -97,7 +97,7 @@ public static LeafReader wrap(IndexReader reader) throws IOException {
    in = reader;
    in.registerParentReader(this);
    if (reader.leaves().isEmpty()) {
-      metaData = new LeafMetaData(Version.LATEST.major, Version.LATEST, null);
+      metaData = new LeafMetaData(Version.LATEST.major, Version.LATEST, null, false);


ref: apache/lucene@2604381

cpoerschke · 2024-01-09T17:07:01Z

solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java

+              createdVersionMajor,
+              minVersion,
+              null,
+              reader.leaves().get(0).reader().getMetaData().hasBlocks());


minor/subjective: reader.leaves().get(0).reader().getMetaData() assignment to a local variable so that both the createdVersionMajor compute above and the hasBlock determination here can use it

cpoerschke · 2024-01-09T17:39:17Z

TestSolrQueryParser#testSyntax - re-run it a few times locally , surprisingly it passed then failed, passed, failed; somewhat flaky.

Was able to reproduce the failure locally with ./gradlew test --tests TestSolrQueryParser.testSyntax -Dtests.seed=9EEE6CC2A6BE4FEA from the CI run. Added a commit to account for the apache/lucene@690462c change and now it passes locally.

cpoerschke · 2024-01-09T17:39:45Z

solr/core/src/test/org/apache/solr/search/TestSolrQueryParser.java

-        expected = "(foo_dt:[1378857600000 TO 1378857600000])";
+        expected = "foo_dt:[1378857600000 TO 1378857600000]";
        if (foo_dt.hasDocValues() && foo_dt.indexed()) {
-          expected = "IndexOrDocValuesQuery" + expected;
+          expected =
+              "IndexOrDocValuesQuery(IndexOrDocValuesQuery(indexQuery="
+                  + expected
+                  + ", dvQuery="
+                  + expected
+                  + "))";
+        } else {
+          expected = "(" + expected + ")";


ref: apache/lucene@690462c

psalagnac · 2024-01-10T14:06:28Z

solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java

+      LeafMetaData leafMetaData = reader.leaves().get(0).reader().getMetaData();
+      metaData =
+          new LeafMetaData(
+              leafMetaData.getCreatedVersionMajor(), minVersion, null, leafMetaData.hasBlocks());


Out of curiosity, why don't we also specify the sort parameter here (3rd parameter)? It could be retrieved from existing leaf metadata (this is not new with this change).

good point, we could specify leafMetaData.getSort() if docs are in no particular order, it would be null.

Out of curiosity, why don't we also specify the sort parameter here (3rd parameter)? It could be retrieved from existing leaf metadata (this is not new with this change).

I wonder if this might be because we don't yet have 'full' index sorting directly configurable in Solr -- https://issues.apache.org/jira/browse/SOLR-13681 exists for that -- i.e. commonly the sort would indeed be null except when a SortingMergePolicy is configured in which case we'd have 'partial' index sorting i.e. some segments could be sorted and some could be unsorted and thus it would be tricky to confidently set the sort value to anything other than the null value.

... (this is not new with this change).

My preference would be to not change the parameter setting as part of the Lucene upgrade but to perhaps leave a note on SOLR-13681 or have a separate issue for that change.

cpoerschke · 2024-01-16T13:34:04Z

cross-referencing w.r.t. the TestScoreJoinQPNoScore#testRandomJoin failures:

https://lists.apache.org/thread/ms65gkmm86cpp5ntdo555ob8mrgg64o5 thread on the Lucene Dev mailing list - thanks @NazerkeBS for creating it.
join: repeat BytesRefHash.sort() in TermsQuery after TermsIncludingScoreQuery lucene#13018 issue opened based on it.

cpoerschke · 2024-01-25T15:08:37Z

FYI I've tried out the Solr-with-local-Lucene documentation steps -- #2223 tweaks are a side effect -- to confirm and have confirmed that apache/lucene#13014 solves the TestScoreJoinQPNoScore#testRandomJoin failures.

./gradlew test --tests TestScoreJoinQPNoScore.testRandomJoin -Plucene.dev.version=9.9.2-SNAPSHOT

NazerkeBS · 2024-01-25T15:43:50Z

FYI I've tried out the Solr-with-local-Lucene documentation steps -- #2223 tweaks are a side effect -- to confirm and have confirmed that apache/lucene#13014 solves the TestScoreJoinQPNoScore#testRandomJoin failures.
./gradlew test --tests TestScoreJoinQPNoScore.testRandomJoin -Plucene.dev.version=9.9.2-SNAPSHOT

Thanks Christine for quickly testing it. I will wait for the v.9.9.2 official release and update this PR.

NazerkeBS · 2024-01-29T14:32:27Z

all tests passed locally

gerlowskija · 2024-01-30T12:38:23Z

This looks awesome, thanks to all! I'm going to test this out locally and then plan to merge later today!

Co-authored-by: Nazerke Seidan <nseidan@Nazerkes-MacBook-Pro.local> Co-authored-by: Christine Poerschke <cpoerschke@apache.org>

cpoerschke · 2024-01-30T14:15:18Z

... I'm going to test this out locally ...

Ran tests locally too, they passed.

Co-authored-by: Nazerke Seidan <nseidan@Nazerkes-MacBook-Pro.local> Co-authored-by: Christine Poerschke <cpoerschke@apache.org>

SOLR-17097: Upgrade Solr to use Lucene 9.9.1

35ca6aa

cpoerschke mentioned this pull request Jan 5, 2024

Update org.apache.lucene:* to v9.9.1 #2168

Closed

1 task

cpoerschke reviewed Jan 9, 2024

View reviewed changes

adjust TestSolrQueryParser.testSyntax

59bf17a

cpoerschke reviewed Jan 9, 2024

View reviewed changes

Update

1cb331b

github-actions bot added dependencies Dependency upgrades configs test-framework labels Jan 10, 2024

psalagnac reviewed Jan 10, 2024

View reviewed changes

use leafMetadata's sort

5f948ce

cpoerschke mentioned this pull request Jan 25, 2024

join: avoid repeat BytesRefHash.sort() in TermsQuery after TermsIncludingScoreQuery apache/lucene#13019

Draft

NazerkeBS changed the title ~~SOLR-17097: Upgrade Solr to use Lucene 9.9.1~~ SOLR-17097: Upgrade Solr to use Lucene 9.9.2 Jan 29, 2024

Update Lucene to 9.9.2

7fb3b41

gerlowskija added 2 commits January 30, 2024 07:39

Merge branch 'main' into lucene991

6e4c2ed

Add CHANGES.txt entry

db8cc69

gerlowskija merged commit 4d22f0c into apache:main Jan 30, 2024
2 of 3 checks passed

gerlowskija pushed a commit that referenced this pull request Jan 30, 2024

SOLR-17097: Upgrade Solr to use Lucene 9.9.2 (#2176)

1bf0de4

Co-authored-by: Nazerke Seidan <nseidan@Nazerkes-MacBook-Pro.local> Co-authored-by: Christine Poerschke <cpoerschke@apache.org>

gerlowskija pushed a commit that referenced this pull request Jan 30, 2024

SOLR-17097: Upgrade Solr to use Lucene 9.9.2 (#2176)

2eed968

Co-authored-by: Nazerke Seidan <nseidan@Nazerkes-MacBook-Pro.local> Co-authored-by: Christine Poerschke <cpoerschke@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SOLR-17097: Upgrade Solr to use Lucene 9.9.2 #2176

SOLR-17097: Upgrade Solr to use Lucene 9.9.2 #2176

NazerkeBS commented Jan 4, 2024 •

edited

epugh commented Jan 4, 2024 •

edited

NazerkeBS commented Jan 4, 2024

cpoerschke Jan 9, 2024

cpoerschke Jan 9, 2024

cpoerschke Jan 9, 2024

cpoerschke commented Jan 9, 2024

cpoerschke Jan 9, 2024

psalagnac Jan 10, 2024

NazerkeBS Jan 11, 2024

cpoerschke Jan 30, 2024

cpoerschke commented Jan 16, 2024

cpoerschke commented Jan 25, 2024

NazerkeBS commented Jan 25, 2024

NazerkeBS commented Jan 29, 2024

gerlowskija commented Jan 30, 2024

cpoerschke commented Jan 30, 2024

		FieldComparator<?> comparator =
		sortField.getComparator(1, Pruning.GREATER_THAN_OR_EQUAL_TO);

SOLR-17097: Upgrade Solr to use Lucene 9.9.2 #2176

SOLR-17097: Upgrade Solr to use Lucene 9.9.2 #2176

Conversation

NazerkeBS commented Jan 4, 2024 • edited

Description

Solution

Tests

Checklist

epugh commented Jan 4, 2024 • edited

NazerkeBS commented Jan 4, 2024

cpoerschke Jan 9, 2024

Choose a reason for hiding this comment

cpoerschke Jan 9, 2024

Choose a reason for hiding this comment

cpoerschke Jan 9, 2024

Choose a reason for hiding this comment

cpoerschke commented Jan 9, 2024

cpoerschke Jan 9, 2024

Choose a reason for hiding this comment

psalagnac Jan 10, 2024

Choose a reason for hiding this comment

NazerkeBS Jan 11, 2024

Choose a reason for hiding this comment

cpoerschke Jan 30, 2024

Choose a reason for hiding this comment

cpoerschke commented Jan 16, 2024

cpoerschke commented Jan 25, 2024

NazerkeBS commented Jan 25, 2024

NazerkeBS commented Jan 29, 2024

gerlowskija commented Jan 30, 2024

cpoerschke commented Jan 30, 2024

NazerkeBS commented Jan 4, 2024 •

edited

epugh commented Jan 4, 2024 •

edited