OPENNLP-1885: Add SentencePiece subword add-on - #178
Draft
krickert wants to merge 8 commits into
Draft
Conversation
Publish add-on artifacts under org.apache.opennlp.addons while keeping core dependencies under org.apache.opennlp. Replace the discussion ledger with a concise proposal and DEV thread references. Red evidence: the canary effective groupId was org.apache.opennlp for the root and each module.
# Conflicts: # src/site/markdown/addon-guidelines.md
Author
|
So this is our "canary" PR that would go into our "addons" - of course we'll need to setup the infra for the whole addons, but this would be the impl for the OPENNLP-1885. This will unlock a lot of the embedding add-ons that are being proposed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SubwordTokenizerAPI with pure Java SentencePiece inference.modelfiles and supports unigram and byte-pair modelsNo runtime model, native library, or third-party training data is bundled.
This is the example add-on contribution described by #177.
Dependencies
SubwordTokenizer,SubwordPiece, and the shared offset contractThe PR targets
OPENNLP-1924-canary-addon. It remains a draft until thecore API dependency is available to the add-ons build.
Validation
mvn clean verify -Dopennlp.forkCount=1: all seven reactor projects passedmvn -N clean site: 9 HTML pages generatedserialvervalues