SOLR-16442: upgrade to Lucene 9.4 (do and document) #1056

cpoerschke · 2022-10-07T17:00:48Z

https://issues.apache.org/jira/browse/SOLR-16442

Collaboration welcome e.g. open questions:

what to do about the sortMissingLast logic in the SortableBinaryField class?
what documentation needs changes for a Lucene dependency upgrade?
in SchemaCodecFactory lucene93 vs. lucene94 use?
properly read https://lucene.apache.org/core/9_4_0/MIGRATE.html

solr/test-framework/src/java/org/apache/solr/schema/SortableBinaryField.java

cpoerschke · 2022-10-07T17:06:33Z

solr/core/src/java/org/apache/solr/core/MMapDirectoryFactory.java

- *       LockFactory, int)}
+ *       LockFactory, long)}
 * </ul>
 */
 public class MMapDirectoryFactory extends StandardDirectoryFactory {
  private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
  boolean unmapHack;
  boolean preload;
-  private int maxChunk;
+  private long maxChunk;

  @Override
  public void init(NamedList<?> args) {
    super.init(args);
    SolrParams params = args.toSolrParams();
-    maxChunk = params.getInt("maxChunkSize", MMapDirectory.DEFAULT_MAX_CHUNK_SIZE);
+    maxChunk = params.getLong("maxChunkSize", MMapDirectory.DEFAULT_MAX_CHUNK_SIZE);


https://github.com/apache/lucene/blob/releases/lucene/9.4.0/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L117-L139

solr/core/src/java/org/apache/solr/core/SchemaCodecFactory.java

cpoerschke · 2022-10-07T17:10:35Z

solr/core/src/java/org/apache/solr/schema/SchemaField.java

+  @Override
+  public VectorEncoding vectorEncoding() {
+    return VectorEncoding.BYTE;
+  }
+


https://github.com/apache/lucene/blob/releases/lucene/9.4.0/lucene/core/src/java/org/apache/lucene/index/IndexableFieldType.java#L104-L105

cpoerschke · 2022-10-07T17:11:30Z

solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java

@@ -522,6 +522,7 @@ private static class ReaderWrapper extends FilterLeafReader {
                  fieldInfo.getPointIndexDimensionCount(),
                  fieldInfo.getPointNumBytes(),
                  fieldInfo.getVectorDimension(),
+                  fieldInfo.getVectorEncoding(),


https://github.com/apache/lucene/blob/releases/lucene/9.4.0/lucene/core/src/java/org/apache/lucene/index/FieldInfo.java#L65-L86

cpoerschke · 2022-10-07T17:15:04Z

solr/test-framework/src/java/org/apache/solr/schema/SortableBinaryField.java

-import org.apache.lucene.search.FieldComparator;
 import org.apache.lucene.search.FieldComparatorSource;
 import org.apache.lucene.search.SortField;
+import org.apache.lucene.search.comparators.TermOrdValComparator;


apache/lucene@261db55

solr/test-framework/src/java/org/apache/solr/schema/SortableBinaryField.java

…naryField.java

sonatype-lift · 2022-10-07T17:50:23Z

⚠️ 314 God Classes were detected by Lift in this project. Visit the Lift web console for more details.

epugh

LGTM. One thought on the docs, would it make sense to reference back to this PR and SOLR-16442 for the next person to see an example? For repetitive tasks, it's great to have a nice model to follow that shows all the types of changes required....

dsmiley · 2022-10-12T20:39:55Z

dev-docs/lucene-upgrade.md

+```
+
+## Code
+


Lets read upgrading notes provided by Lucene first; ehh?
https://lucene.apache.org/core/9_4_0/MIGRATE.html

cpoerschke · 2022-10-19T09:00:38Z

LGTM. One thought on the docs, would it make sense to reference back to this PR and SOLR-16442 for the next person to see an example? For repetitive tasks, it's great to have a nice model to follow that shows all the types of changes required....

I'd broken up the change into distinctive commits with that sort of use and potentially review ease in mind.

In terms of explicitly referencing back to the PR and/or JIRA issue, I'm not so convinced on that for two reasons:

ideally the documentation should mention all that needs doing i.e. for the upgrade to be done based on documentation rather than past example(s)
it might be off-putting or restrictive to see 13 commits i.e. working through a documented list incrementally is one thing but perceiving that things should be done in separate commits might not be everyone's cup-of-tea

risdenk · 2022-10-19T09:30:10Z

dev-docs/lucene-upgrade.md

+### `solr/licenses` update
+
+```
+git rm lucene-*-9.3.0.jar.sha1


./gradlew updateLicenses will handle this

thanks! #1087 to simplify the docs (and I'll leave that PR open for a few days in case of additional documentation feedback)

(cherry picked from commit 4ab7f4b) Resolved Conflicts: solr/solr-ref-guide/antora.yml

cpoerschke added 8 commits October 7, 2022 16:58

SOLR-16442: start dev-docs/lucene-upgrade.md

b95a32a

versions.props update

5c991e6

versions.lock update

47ce4ef

solr/licenses update

5103150

compileJava

9f73dbd

compileTestJava

6539a45

precommit

ca2895b

test

46c7d3b

cpoerschke commented Oct 7, 2022

View reviewed changes

solr/test-framework/src/java/org/apache/solr/schema/SortableBinaryField.java Outdated Show resolved Hide resolved

cpoerschke commented Oct 7, 2022

View reviewed changes

Update solr/test-framework/src/java/org/apache/solr/schema/SortableBi…

a07738f

…naryField.java

cpoerschke marked this pull request as ready for review October 7, 2022 17:34

documentation: solr/solr-ref-guide/antora.yml

6368948

lucene-upgrade.md: 'codec changes' --> 'codec changes (if any)'

119f03a

epugh approved these changes Oct 12, 2022

View reviewed changes

dsmiley reviewed Oct 12, 2022

View reviewed changes

add 'read' as first step

3685857

cpoerschke marked this pull request as draft October 14, 2022 17:18

dsmiley marked this pull request as ready for review October 15, 2022 03:21

dsmiley approved these changes Oct 15, 2022

View reviewed changes

Merge branch 'main' into lucene940

213ec64

cpoerschke merged commit 4ab7f4b into apache:main Oct 19, 2022

cpoerschke deleted the lucene940 branch October 19, 2022 09:25

risdenk reviewed Oct 19, 2022

View reviewed changes

asfgit pushed a commit that referenced this pull request Oct 19, 2022

SOLR-16442: upgrade to Lucene 9.4 (do and document) (#1056)

795b294

(cherry picked from commit 4ab7f4b) Resolved Conflicts: solr/solr-ref-guide/antora.yml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SOLR-16442: upgrade to Lucene 9.4 (do and document) #1056

SOLR-16442: upgrade to Lucene 9.4 (do and document) #1056

cpoerschke commented Oct 7, 2022 •

edited

cpoerschke Oct 7, 2022

cpoerschke Oct 7, 2022

cpoerschke Oct 7, 2022

cpoerschke Oct 7, 2022

sonatype-lift bot commented Oct 7, 2022

epugh left a comment

dsmiley Oct 12, 2022

cpoerschke commented Oct 19, 2022

risdenk Oct 19, 2022

cpoerschke Oct 19, 2022

		```

		## Code

SOLR-16442: upgrade to Lucene 9.4 (do and document) #1056

SOLR-16442: upgrade to Lucene 9.4 (do and document) #1056

Conversation

cpoerschke commented Oct 7, 2022 • edited

cpoerschke Oct 7, 2022

Choose a reason for hiding this comment

cpoerschke Oct 7, 2022

Choose a reason for hiding this comment

cpoerschke Oct 7, 2022

Choose a reason for hiding this comment

cpoerschke Oct 7, 2022

Choose a reason for hiding this comment

sonatype-lift bot commented Oct 7, 2022

epugh left a comment

Choose a reason for hiding this comment

dsmiley Oct 12, 2022

Choose a reason for hiding this comment

cpoerschke commented Oct 19, 2022

risdenk Oct 19, 2022

Choose a reason for hiding this comment

cpoerschke Oct 19, 2022

Choose a reason for hiding this comment

cpoerschke commented Oct 7, 2022 •

edited