LUCENE-9684: Hunspell: support COMPOUNDRULE #2228
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Reviewing and merging commits separately might be a good idea. |
e.g. for English ordinal numbers
otherwise dictionaries with long/number flags with all-caps words are broken
When checking an upper- or title-cased word, consider its case variants like Stemmer does
dweiss
reviewed
Jan 22, 2021
| import org.apache.lucene.util.IntsRef; | ||
|
|
||
| class CompoundRule { | ||
| private final char[] data; |
There was a problem hiding this comment.
I wonder if keeping char[] instead of the string itself buys us anything here. Strings can use more efficient storage than on newer JVMs (compact strings) and using charAt instead of array accesses is pretty much the same?
There was a problem hiding this comment.
Memory shouldn't be of much concern, as AFAIK there are usually very few rules. Array access is a bit shorter in code, so I'd prefer to leave it.
dweiss
approved these changes
Jan 22, 2021
Sure, I did :) |
|
How do we proceed here? Should anyone else review this? |
|
No, I'll commit it in. I have an unrelated day job so expect some delays. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Hunspell uses COMPOUNDRULE e.g. for ordinal numbers in en-US dictionary (1st, 42nd, etc)
Solution
COMPOUNDRULE is a regexp-like pattern over word parts flags. I've reimplemented Hunspell's logic, which breaks the word into parts in different ways and checks whether any COMPOUNDRULE matches them (commit 1). To support uppercase, I had also to repeat
Stemmer's case variations (commit 3). While doing this, I discovered a bug in all-caps treatment where HIDDEN flag didn't play well with non-single-character flag formats, which I fixed (commit 2).Tests
All
compoundrule*tests taken from Hunspell repository, plus a randomized test for flag serialization and deserialization.Checklist
Please review the following and check all that apply:
masterbranch../gradlew check.