Skip to content

SOLR-17023: Use Modern NLP Models via ONNX and Apache OpenNLP with Solr #1999

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 42 commits into
base: main
Choose a base branch
from
Open

SOLR-17023: Use Modern NLP Models via ONNX and Apache OpenNLP with Solr #1999

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
f19f660
first version w bats test
epugh Oct 10, 2023
48fd37b
not positive I need this
epugh Oct 10, 2023
cecd4f6
new name, and OpenNLP is kind of a implmentation detail ;-).
epugh Oct 10, 2023
10abbd0
baby steps, found that packagesstore blows up killing solr when i pos…
epugh Oct 10, 2023
dd4cc89
remove resolutionStraregy force from gradle build..
epugh Oct 10, 2023
af18d0c
match name in the underlying OpenNLP project. bikeshedding!
epugh Oct 10, 2023
6950629
tidy
epugh Oct 10, 2023
1adba22
reorder params
epugh Oct 11, 2023
bcbf16d
log formatting
epugh Oct 11, 2023
e634b1c
regenerate...
epugh Oct 11, 2023
778e3f4
dynamically grab the models from hugging face.
epugh Oct 11, 2023
405153b
use logging structure for stack traces
epugh Oct 11, 2023
4e54090
download the correct jar, and document the work to remove this need i…
epugh Oct 13, 2023
9e30c29
Upgrade to OpenNLP 2.3.1 and related dependencies.
epugh Nov 27, 2023
ef9364a
no longer need workarounds for gpu/cpu issues with updated OpenNLP.
epugh Nov 27, 2023
08b3de8
Merge remote-tracking branch 'upstream/main' into SOLR-17023
epugh Nov 27, 2023
2c106e9
We cleaned up the name ;-)
epugh Dec 5, 2023
8112229
Merge branch 'main' into SOLR-17023
epugh Jan 9, 2024
8d6187f
prompted to update the locks
epugh Jan 9, 2024
50887bb
Add in required license files
epugh Jan 9, 2024
a31ef8e
lint
epugh Jan 9, 2024
4b7528a
precommit warning
epugh Jan 9, 2024
3a760da
Merge remote-tracking branch 'origin/main' into SOLR-17023
cpoerschke Feb 13, 2024
bdee568
upgrade OpenNLP from 2.3.1 to 2.3.2 (to match Lucene main branch)
cpoerschke Feb 13, 2024
541dceb
tentative: minimum Java17 for this PR
cpoerschke Feb 13, 2024
633391e
Update gradle-precommit.yml - Java 11 --> 17
cpoerschke Feb 13, 2024
e2fe1fe
Update solrj-test.yml - Java 11 --> 17
cpoerschke Feb 13, 2024
855b5ea
Update docker-test.yml - Java 11 --> 17
cpoerschke Feb 13, 2024
9c16a8b
Update bin-solr-test.yml - Java 11 --> 17
cpoerschke Feb 13, 2024
476a7d4
undo 'tentative: minimum Java17 for this PR' -- see PR 1510 instead
cpoerschke Feb 13, 2024
86dd0aa
Merge remote-tracking branch 'upstream/main' into SOLR-17023
epugh Feb 23, 2025
8bffd15
Migrate to new way of specifying dependency versions
epugh Feb 23, 2025
f374b4a
New home for FILESTORE_DIRECTORY constant
epugh Feb 23, 2025
36fbb39
lint
epugh Feb 23, 2025
c623363
Add Onnx to list of dependencies
epugh Feb 24, 2025
354653d
Update test for recent Solr changes.
epugh Feb 24, 2025
e61296f
Satisfy linking warning
epugh Feb 24, 2025
85d6e16
Merge remote-tracking branch 'upstream/main' into SOLR-17023
epugh May 27, 2025
84905fe
Update for latest file pattern
epugh Jun 13, 2025
dd274ff
Merge remote-tracking branch 'upstream/main' into SOLR-17023
epugh Jun 13, 2025
a93b5db
Update to latest OpenNLP, also used by Lucene.
epugh Jun 13, 2025
74e0132
One more dependency declaration.
epugh Jun 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion gradle/libs.versions.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ apache-httpcomponents-httpmime = "4.5.14"
apache-kafka = "3.9.0"
apache-log4j = "2.21.0"
apache-lucene = "9.12.1"
apache-opennlp = "1.9.4"
apache-opennlp = "2.5.4"
apache-poi = "5.2.2"
apache-rat = "0.15"
apache-tika = "1.28.5"
Expand Down Expand Up @@ -168,6 +168,7 @@ nimbusds-josejwt = "9.48"
nodegradle-node = "7.0.1"
# @keep Node JS version used in node.gradle (LTS)
nodejs = "16.20.2"
onnx = "1.21.0"
openapi = "7.6.0"
openjdk-jmh = "1.37"
opentelemetry = "1.46.0"
Expand Down Expand Up @@ -280,6 +281,7 @@ apache-lucene-queryparser = { module = "org.apache.lucene:lucene-queryparser", v
apache-lucene-spatialextras = { module = "org.apache.lucene:lucene-spatial-extras", version.ref = "apache-lucene" }
apache-lucene-suggest = { module = "org.apache.lucene:lucene-suggest", version.ref = "apache-lucene" }
apache-lucene-testframework = { module = "org.apache.lucene:lucene-test-framework", version.ref = "apache-lucene" }
apache-opennlp-dl = { module = "org.apache.opennlp:opennlp-dl", version.ref = "apache-opennlp" }
apache-opennlp-tools = { module = "org.apache.opennlp:opennlp-tools", version.ref = "apache-opennlp" }
apache-poi-ooxml = { module = "org.apache.poi:poi-ooxml", version.ref = "apache-poi" }
apache-poi-poi = { module = "org.apache.poi:poi", version.ref = "apache-poi" }
Expand Down Expand Up @@ -465,6 +467,7 @@ netty-transport-classes-epoll = { module = "io.netty:netty-transport-classes-epo
# @keep transitive dependency for version alignment
netty-transport-native-epoll = { module = "io.netty:netty-transport-native-epoll", version.ref = "netty" }
nimbusds-josejwt = { module = "com.nimbusds:nimbus-jose-jwt", version.ref = "nimbusds-josejwt" }
onnxruntime = { module = "com.microsoft.onnxruntime:onnxruntime", version.ref = "onnx" }
openjdk-jmh-core = { module = "org.openjdk.jmh:jmh-core", version.ref = "openjdk-jmh" }
openjdk-jmh-generatorannprocess = { module = "org.openjdk.jmh:jmh-generator-annprocess", version.ref = "openjdk-jmh" }
opentelemetry-api = { module = "io.opentelemetry:opentelemetry-api", version.ref = "opentelemetry" }
Expand Down
7 changes: 5 additions & 2 deletions solr/core/gradle.lockfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ com.google.protobuf:protobuf-java:3.25.3=annotationProcessor,errorprone,testAnno
com.ibm.icu:icu4j:74.2=jarValidation,testRuntimeClasspath
com.j256.simplemagic:simplemagic:1.17=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
com.jayway.jsonpath:json-path:2.9.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
com.microsoft.onnxruntime:onnxruntime:1.21.0=jarValidation,testRuntimeClasspath
com.tdunning:t-digest:3.3=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
commons-cli:commons-cli:1.9.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
commons-codec:commons-codec:1.17.1=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
Expand Down Expand Up @@ -121,7 +122,8 @@ org.apache.lucene:lucene-spatial-extras:9.12.1=compileClasspath,jarValidation,ru
org.apache.lucene:lucene-spatial3d:9.12.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
org.apache.lucene:lucene-suggest:9.12.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
org.apache.lucene:lucene-test-framework:9.12.1=jarValidation,testCompileClasspath,testRuntimeClasspath
org.apache.opennlp:opennlp-tools:1.9.4=jarValidation,testRuntimeClasspath
org.apache.opennlp:opennlp-dl:2.5.4=jarValidation,testRuntimeClasspath
org.apache.opennlp:opennlp-tools:2.5.4=jarValidation,testRuntimeClasspath
org.apache.zookeeper:zookeeper-jute:3.9.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
org.apache.zookeeper:zookeeper:3.9.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
org.apiguardian:apiguardian-api:1.1.2=jarValidation,testCompileClasspath,testRuntimeClasspath
Expand Down Expand Up @@ -193,7 +195,8 @@ org.ow2.asm:asm:9.7.1=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClas
org.pcollections:pcollections:4.0.1=annotationProcessor,errorprone,testAnnotationProcessor
org.semver4j:semver4j:5.3.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.slf4j:jcl-over-slf4j:2.0.16=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.slf4j:slf4j-api:2.0.16=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
org.slf4j:slf4j-api:2.0.16=apiHelper,compileClasspath,runtimeClasspath,runtimeLibs,testCompileClasspath
org.slf4j:slf4j-api:2.0.17=jarValidation,testRuntimeClasspath
org.xerial.snappy:snappy-java:1.1.10.5=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
ua.net.nlp:morfologik-ukrainian-search:4.9.1=jarValidation,testRuntimeClasspath
empty=apiHelperTest,missingdoclet,permitAggregatorUse,permitTestAggregatorUse,permitTestUnusedDeclared,permitTestUsedUndeclared,permitUsedUndeclared,signatures
1 change: 1 addition & 0 deletions solr/licenses/onnxruntime-1.21.0.jar.sha1
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
43ee677ed42fbe2693e2e052307d48df754d74cf
21 changes: 21 additions & 0 deletions solr/licenses/onnxruntime-LICENSE-MIT.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking up https://github.com/microsoft/onnxruntime/blob/v1.15.0/LICENSE here I noticed there's a 1.15.1 and 1.16.x now too, wondering about 1.15.0 vs. the later ones then.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

part-answering own question: OpenNLP 2.3.1 uses onnxruntime 1.15.0 - https://github.com/apache/opennlp/blame/opennlp-2.3.1/pom.xml#L176

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't specify the version of onnx, so maybe we poke them to update?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Next opennlp will have 1.16.3 (or higher)


Copyright (c) Microsoft Corporation

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Loading
Loading