HyphenationCompoundWordTokenFilter creates overlapping tokens with onlyLongestMatch enabled [LUCENE-8183] #9231
Comments
|
Rupert Westenthaler (migrated from JIRA) Added a patch that shows how the unexpected behaviour could be fixed. |
|
Uwe Schindler (@uschindler) (migrated from JIRA) Hi, |
|
Matthias Krueger (@mkr) (migrated from JIRA) Rupert Westenthaler Quick question regarding your patch: What's the reasoning behind not decomposing terms that are part of the dictionary at all? The Example: I like the idea of restricting this further to only the longest terms that touch a certain hyphenation point. This would exclude "schaft" in the example above (as "Wirtschaft" and "wissenschaft" are two longer terms encompassing the respective hyphenation point). On the other hand, there might be examples where you still want to include the "overlapping" tokens. For "Fußballpumpe" and dictionary Regarding point 1: The current algorithm always returns the term itself again if it's part of the dictionary. I guess, this could be changed if we don't check against Perhaps these kind of adjustments should rather be done in a TokenFilter similar to RemoveDuplicatesTokenFilter instead of complicating the decompounding algorithm? |
|
Rupert Westenthaler (migrated from JIRA) AFAIK I use exactly this dictionary and hyphenator config. I will provide a Solr core config that can be used to reproduce the described issue. |
|
Rupert Westenthaler (migrated from JIRA) I was not aware that this is the intended behaviour.
Thats a good point. Maybe one should still consider parts that are not enclosed by an token that was already decomposed. So for This reminds me to
I am aware of this possibility. In fact I do use the |
|
Uwe Schindler (@uschindler) (migrated from JIRA) Rupert Westenthaler: I was not aware that this was my dictionary file! The names in your example (under "environment in your report) did not look like the example listed here: https://github.com/uschindler/german-decompounder |
|
Uwe Schindler (@uschindler) (migrated from JIRA)
I think it's good to not add them in the first place. The change is quite simple, so it can be done here. And it does not really complicate the algorithm as its done at one separated place. |
|
Uwe Schindler (@uschindler) (migrated from JIRA) I have to check the algorithm, but to make this patch into lucene, the test cases need to be adapted to check this new behaviour. |
|
Rupert Westenthaler (migrated from JIRA) FYI: I pan to spend some time to implement a version of the DictionaryCompoundWordTokenFilter that adds options for
IMO the simplest way is to first emit all tokens and later filter those based on the active options ( Regarding the test: Providing good test examples is hard as the current test cases are based on a Danish and I do not speak this language |
|
Matthias Krueger (@mkr) (migrated from JIRA) You might want to have a look at "mocking" the HyphenationTree (see my patch for LUCENE-8185) which simplifies writing a decompounding test. |
|
Rupert Westenthaler (migrated from JIRA) Thats helpful indeed! thx |
|
Rupert Westenthaler (migrated from JIRA) Patch: LUCENE-8183_20180227_rwesten.diff New Parameters:
together with the existing Typically users will only want to include one of the three attributes as enabling Because of that it would be an option to refactor this to an single attribute with different setting, but this would require to think about backward compatibility for configurations that do use AlgorithmIf processing of subWords is deactivated (any of I changed the iteration direction of the inner NOTE: that this also changes the order of the Tokens in the token stream but as all tokens are at the same position that should not make any difference. I had however to modify some existing tests as those where sensitive to the ordering h3 Tests I added two test methods in
In addition h3 Environment This patch is based on LUCENE-8183_20180227_rwesten.diffdiff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/HyphenationCompoundWordTokenFilter.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/HyphenationCompoundWordTokenFilter.java
index 41f92c9..ccb8714 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/HyphenationCompoundWordTokenFilter.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/HyphenationCompoundWordTokenFilter.java
@@ -34,6 +34,9 @@ import org.xml.sax.InputSource;
public class HyphenationCompoundWordTokenFilter extends
CompoundWordTokenFilterBase {
private HyphenationTree hyphenator;
+ private boolean noSubMatches;
+ private boolean noOverlappingMatches;
+ private boolean calcSubMatches;
/**
* Creates a new {@link HyphenationCompoundWordTokenFilter} instance.
@@ -48,7 +51,7 @@ public class HyphenationCompoundWordTokenFilter extends
public HyphenationCompoundWordTokenFilter(TokenStream input,
HyphenationTree hyphenator, CharArraySet dictionary) {
this(input, hyphenator, dictionary, DEFAULT_MIN_WORD_SIZE,
- DEFAULT_MIN_SUBWORD_SIZE, DEFAULT_MAX_SUBWORD_SIZE, false);
+ DEFAULT_MIN_SUBWORD_SIZE, DEFAULT_MAX_SUBWORD_SIZE, false, false, false);
}
/**
@@ -67,21 +70,54 @@ public class HyphenationCompoundWordTokenFilter extends
* @param maxSubwordSize
* only subwords shorter than this get to the output stream
* @param onlyLongestMatch
- * Add only the longest matching subword to the stream
+ * Add only the longest matching subword for each hyphenation to the stream
+ */
+ public HyphenationCompoundWordTokenFilter(TokenStream input,
+ HyphenationTree hyphenator, CharArraySet dictionary, int minWordSize,
+ int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch) {
+ this(input, hyphenator, dictionary, minWordSize, minSubwordSize,
+ maxSubwordSize, onlyLongestMatch, false, false);
+ }
+
+ /**
+ * Creates a new {@link HyphenationCompoundWordTokenFilter} instance.
+ *
+ * @param input
+ * the {@link org.apache.lucene.analysis.TokenStream} to process
+ * @param hyphenator
+ * the hyphenation pattern tree to use for hyphenation
+ * @param dictionary
+ * the word dictionary to match against.
+ * @param minWordSize
+ * only words longer than this get processed
+ * @param minSubwordSize
+ * only subwords longer than this get to the output stream
+ * @param maxSubwordSize
+ * only subwords shorter than this get to the output stream
+ * @param onlyLongestMatch
+ * Add only the longest matching subword for each hyphenation to the stream
+ * @param noSubMatches
+ * Excludes subwords that are enclosed by an other token
+ * @param noOverlappingMatches
+ * Excludes subwords that overlap with an other subword
*/
public HyphenationCompoundWordTokenFilter(TokenStream input,
HyphenationTree hyphenator, CharArraySet dictionary, int minWordSize,
- int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch) {
+ int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch,
+ boolean noSubMatches, boolean noOverlappingMatches) {
super(input, dictionary, minWordSize, minSubwordSize, maxSubwordSize,
onlyLongestMatch);
this.hyphenator = hyphenator;
+ this.noSubMatches = noSubMatches;
+ this.noOverlappingMatches = noOverlappingMatches;
+ this.calcSubMatches = !onlyLongestMatch && !noSubMatches && !noOverlappingMatches;
}
/**
* Create a HyphenationCompoundWordTokenFilter with no dictionary.
* <p>
- * Calls {@link #HyphenationCompoundWordTokenFilter(org.apache.lucene.analysis.TokenStream, org.apache.lucene.analysis.compound.hyphenation.HyphenationTree, org.apache.lucene.analysis.CharArraySet, int, int, int, boolean)
+ * Calls {@link #HyphenationCompoundWordTokenFilter(org.apache.lucene.analysis.TokenStream, org.apache.lucene.analysis.compound.hyphenation.HyphenationTree, org.apache.lucene.analysis.CharArraySet, int, int, int, boolean, boolean, boolean)
* HyphenationCompoundWordTokenFilter(matchVersion, input, hyphenator,
* null, minWordSize, minSubwordSize, maxSubwordSize }
*/
@@ -89,7 +125,7 @@ public class HyphenationCompoundWordTokenFilter extends
HyphenationTree hyphenator, int minWordSize, int minSubwordSize,
int maxSubwordSize) {
this(input, hyphenator, null, minWordSize, minSubwordSize,
- maxSubwordSize, false);
+ maxSubwordSize, false, false, false);
}
/**
@@ -133,26 +169,42 @@ public class HyphenationCompoundWordTokenFilter extends
@Override
protected void decompose() {
+ //if the token is in the dictionary and we are not interested in subMatches
+ //we can skip decomposing this token (see testNoSubAndTokenInDictionary unit test)
+ //NOTE:
+ //we check against token and the token that is one character
+ //shorter to avoid problems with genitive 's characters and other binding characters
+ if(dictionary != null && !this.calcSubMatches &&
+ (dictionary.contains(termAtt.buffer(), 0, termAtt.length()) ||
+ termAtt.length() > 1 && dictionary.contains(termAtt.buffer(), 0, termAtt.length() - 1))){
+ return; //the whole token is in the dictionary - do not decompose
+ }
+
// get the hyphenation points
Hyphenation hyphens = hyphenator.hyphenate(termAtt.buffer(), 0, termAtt.length(), 1, 1);
// No hyphen points found -> exit
if (hyphens == null) {
return;
}
+ int maxSubwordSize = Math.min(this.maxSubwordSize, termAtt.length()-1);
final int[] hyp = hyphens.getHyphenationPoints();
+ int consumed = -1; //hyp of the longest token added (for noSub)
+
for (int i = 0; i < hyp.length; ++i) {
- int remaining = hyp.length - i;
+ if(noOverlappingMatches){ //if we do not want overlapping subwords
+ i = Math.max(i, consumed); //skip over consumed hyp
+ }
int start = hyp[i];
- CompoundToken longestMatchToken = null;
- for (int j = 1; j < remaining; j++) {
- int partLength = hyp[i + j] - start;
+ int until = noSubMatches ? Math.max(consumed, i) : i;
+ for (int j = hyp.length - 1; j > until; j--) {
+ int partLength = hyp[j] - start;
// if the part is longer than maxSubwordSize we
// are done with this round
- if (partLength > this.maxSubwordSize) {
- break;
+ if (partLength > maxSubwordSize) {
+ continue;
}
// we only put subwords to the token stream
@@ -160,42 +212,26 @@ public class HyphenationCompoundWordTokenFilter extends
if (partLength < this.minSubwordSize) {
// BOGUS/BROKEN/FUNKY/WACKO: somehow we have negative 'parts' according to the
// calculation above, and we rely upon minSubwordSize being >=0 to filter them out...
- continue;
+ break;
}
// check the dictionary
if (dictionary == null || dictionary.contains(termAtt.buffer(), start, partLength)) {
- if (this.onlyLongestMatch) {
- if (longestMatchToken != null) {
- if (longestMatchToken.txt.length() < partLength) {
- longestMatchToken = new CompoundToken(start, partLength);
- }
- } else {
- longestMatchToken = new CompoundToken(start, partLength);
- }
- } else {
tokens.add(new CompoundToken(start, partLength));
- }
+ consumed = j; //mark the current hyp as consumed
+ if(!calcSubMatches){
+ break; //do not search for shorter matches
+ }
} else if (dictionary.contains(termAtt.buffer(), start, partLength - 1)) {
// check the dictionary again with a word that is one character
- // shorter
- // to avoid problems with genitive 's characters and other binding
- // characters
- if (this.onlyLongestMatch) {
- if (longestMatchToken != null) {
- if (longestMatchToken.txt.length() < partLength - 1) {
- longestMatchToken = new CompoundToken(start, partLength - 1);
- }
- } else {
- longestMatchToken = new CompoundToken(start, partLength - 1);
- }
- } else {
- tokens.add(new CompoundToken(start, partLength - 1));
+ // shorter to avoid problems with genitive 's characters and
+ // other binding characters
+ tokens.add(new CompoundToken(start, partLength - 1));
+ consumed = j; //mark the current hyp as consumed
+ if(!calcSubMatches){
+ break; //do not search for shorter matches
}
- }
- }
- if (this.onlyLongestMatch && longestMatchToken!=null) {
- tokens.add(longestMatchToken);
+ } //else dictionary is present but does not contain the part
}
}
}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/HyphenationCompoundWordTokenFilterFactory.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/HyphenationCompoundWordTokenFilterFactory.java
index 6b018d6..9f6e65c 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/HyphenationCompoundWordTokenFilterFactory.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/HyphenationCompoundWordTokenFilterFactory.java
@@ -69,6 +69,8 @@ public class HyphenationCompoundWordTokenFilterFactory extends TokenFilterFactor
private final int minSubwordSize;
private final int maxSubwordSize;
private final boolean onlyLongestMatch;
+ private final boolean noSubMatches;
+ private final boolean noOverlappingMatches;
/** Creates a new HyphenationCompoundWordTokenFilterFactory */
public HyphenationCompoundWordTokenFilterFactory(Map<String, String> args) {
@@ -80,6 +82,8 @@ public class HyphenationCompoundWordTokenFilterFactory extends TokenFilterFactor
minSubwordSize = getInt(args, "minSubwordSize", CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE);
maxSubwordSize = getInt(args, "maxSubwordSize", CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE);
onlyLongestMatch = getBoolean(args, "onlyLongestMatch", false);
+ noSubMatches = getBoolean(args, "noSubMatches", false);
+ noOverlappingMatches = getBoolean(args, "noOverlappingMatches", false);
if (!args.isEmpty()) {
throw new IllegalArgumentException("Unknown parameters: " + args);
}
@@ -105,6 +109,6 @@ public class HyphenationCompoundWordTokenFilterFactory extends TokenFilterFactor
@Override
public TokenFilter create(TokenStream input) {
- return new HyphenationCompoundWordTokenFilter(input, hyphenator, dictionary, minWordSize, minSubwordSize, maxSubwordSize, onlyLongestMatch);
+ return new HyphenationCompoundWordTokenFilter(input, hyphenator, dictionary, minWordSize, minSubwordSize, maxSubwordSize, onlyLongestMatch, noSubMatches, noOverlappingMatches);
}
}
diff --git a/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/hyphenation/Hyphenation.java b/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/hyphenation/Hyphenation.java
index 3fb1e04..c435eaa 100644
--- a/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/hyphenation/Hyphenation.java
+++ b/lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/hyphenation/Hyphenation.java
@@ -23,12 +23,12 @@ package org.apache.lucene.analysis.compound.hyphenation;
*/
public class Hyphenation {
- private int[] hyphenPoints;
+ private final int[] hyphenPoints;
/**
* rawWord as made of alternating strings and {@link Hyphen Hyphen} instances
*/
- Hyphenation(int[] points) {
+ public Hyphenation(int[] points) {
hyphenPoints = points;
}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/TestCompoundWordTokenFilter.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/TestCompoundWordTokenFilter.java
index 67a1bb4..88ffe8b 100644
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/TestCompoundWordTokenFilter.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/TestCompoundWordTokenFilter.java
@@ -21,6 +21,8 @@ import java.io.IOException;
import java.io.Reader;
import java.io.StringReader;
import java.util.Arrays;
+import java.util.Collections;
+import java.util.Map;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.BaseTokenStreamTestCase;
@@ -31,6 +33,7 @@ import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.Tokenizer;
import org.apache.lucene.analysis.charfilter.MappingCharFilter;
import org.apache.lucene.analysis.charfilter.NormalizeCharMap;
+import org.apache.lucene.analysis.compound.hyphenation.Hyphenation;
import org.apache.lucene.analysis.compound.hyphenation.HyphenationTree;
import org.apache.lucene.analysis.core.KeywordTokenizer;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
@@ -102,7 +105,7 @@ public class TestCompoundWordTokenFilter extends BaseTokenStreamTestCase {
// min=2, max=4
assertTokenStreamContents(tf,
- new String[] { "basketballkurv", "ba", "sket", "bal", "ball", "kurv" }
+ new String[] { "basketballkurv", "ba", "sket", "ball", "bal", "kurv" }
);
tf = new HyphenationCompoundWordTokenFilter(
@@ -126,8 +129,8 @@ public class TestCompoundWordTokenFilter extends BaseTokenStreamTestCase {
// min=4, max=10
assertTokenStreamContents(tf,
- new String[] { "basketballkurv", "basket", "basketbal", "basketball", "sket",
- "sketbal", "sketball", "ball", "ballkurv", "lkurv", "kurv" }
+ new String[] { "basketballkurv", "basketball", "basketbal", "basket",
+ "sketball", "sketbal", "sket", "ballkurv", "ball", "lkurv", "kurv" }
);
}
@@ -273,10 +276,95 @@ public class TestCompoundWordTokenFilter extends BaseTokenStreamTestCase {
"Rindfleisch"),
hyphenator);
- // TODO Rindfleisch returned twice is another issue of the HyphenationCompoundTokenFilter
- assertTokenStreamContents(tf, new String[] { "Rindfleisch", "Rind", "Rindfleisch", "fleisch"});
+ assertTokenStreamContents(tf, new String[] { "Rindfleisch", "Rind", "fleisch"});
}
-
+
+ public void testNoSubAndNoOverlap() throws Exception { //LUCENE-8183
+ String input = "fußballpumpe";
+ Hyphenation hyphenation = new Hyphenation(new int[]{ 0, 3, 7, 10, 12});//fuß ball pum pe
+ HyphenationTree hyphenator = new MockHyphenator(Collections.singletonMap(input, hyphenation));
+ CharArraySet dictionary = makeDictionary("fußball", "ballpumpe", "fuß", "ball", "pumpe");
+
+ //test the default configuration
+ HyphenationCompoundWordTokenFilter tf1 = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary);
+ assertTokenStreamContents(tf1, new String[] { "fußballpumpe", "fußball", "fuß", "ballpumpe", "ball", "pumpe"});
+
+ //test with onlyLongestMatch
+ HyphenationCompoundWordTokenFilter tf2 = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, true);
+ assertTokenStreamContents(tf2, new String[] { "fußballpumpe", "fußball", "ballpumpe", "pumpe"});
+
+ //test with noSub enabled and noOverlap disabled
+ HyphenationCompoundWordTokenFilter tf3 = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, true, true, false);
+ assertTokenStreamContents(tf3, new String[] { "fußballpumpe", "fußball", "ballpumpe"});
+ //assert that the onlyLongestMatch state does not matter if noSub is active
+ HyphenationCompoundWordTokenFilter tf3b = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, false, true, false);
+ assertTokenStreamContents(tf3b, new String[] { "fußballpumpe", "fußball", "ballpumpe"});
+
+ //test with noOverlap enabled
+ HyphenationCompoundWordTokenFilter tf4 = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, true, true, true);
+ //NOTE: 'fußball' consumes 'ball' as possible start so 'ballpumpe' is not considered and 'pumpe' is added
+ assertTokenStreamContents(tf4, new String[] { "fußballpumpe", "fußball", "pumpe"});
+ //assert that the noSub and onlyLongestMatch states do not matter
+ HyphenationCompoundWordTokenFilter tf4b = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, false, false, true);
+ assertTokenStreamContents(tf4b, new String[] { "fußballpumpe", "fußball", "pumpe"});
+ HyphenationCompoundWordTokenFilter tf4c = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, true, false, true);
+ assertTokenStreamContents(tf4c, new String[] { "fußballpumpe", "fußball", "pumpe"});
+ }
+
+ public void testNoSubAndTokenInDictionary() throws Exception { //LUCENE-8183
+ //test that no subwords are added if the token is part of the dictionary and
+ //onlyLongestMatch or noSub is present
+ String input = "fußball";
+ Hyphenation hyphenation = new Hyphenation(new int[]{ 0, 3, 7});//fuß ball
+ HyphenationTree hyphenator = new MockHyphenator(Collections.singletonMap(input, hyphenation));
+ CharArraySet dictionary = makeDictionary("fußball", "fuß", "ball");
+
+ //test the default configuration as baseline
+ HyphenationCompoundWordTokenFilter tf5 = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary);
+ assertTokenStreamContents(tf5, new String[] { "fußball", "fuß", "ball"});
+
+ //when onlyLongestMatch is enabled fußball matches dictionary. So even so
+ //fußball is not added as token it MUST prevent shorter matches to be added
+ HyphenationCompoundWordTokenFilter tf6 = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, true, false, false);
+ assertTokenStreamContents(tf6, new String[] { "fußball"});
+
+ //when noSub is enabled fuß and ball MUST NOT be added as subwords as fußball is in the dictionary
+ HyphenationCompoundWordTokenFilter tf7 = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, false, true, false);
+ assertTokenStreamContents(tf7, new String[] { "fußball"});
+
+ //when noOverlap is enabled fuß and ball MUST NOT be added as subwords as fußball is in the dictionary
+ HyphenationCompoundWordTokenFilter tf8 = new HyphenationCompoundWordTokenFilter(whitespaceMockTokenizer(input),
+ hyphenator, dictionary, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, false, false, true);
+ assertTokenStreamContents(tf8, new String[] { "fußball"});
+}
public static interface MockRetainAttribute extends Attribute {
void setRetain(boolean attr);
@@ -331,7 +419,22 @@ public class TestCompoundWordTokenFilter extends BaseTokenStreamTestCase {
}
}
}
-
+
+ // Hyphenator that has prior knowledge of hyphenation points for terms
+ private static class MockHyphenator extends HyphenationTree {
+
+ private final Map<String, Hyphenation> hyphenations;
+
+ MockHyphenator(Map<String, Hyphenation> hyphenations) {
+ this.hyphenations = hyphenations;
+ }
+
+ @Override
+ public Hyphenation hyphenate(char[] w, int offset, int len, int remainCharCount, int pushCharCount) {
+ return hyphenations.get(new String(w, offset, len));
+ }
+ }
+
// [SOLR-2891](https://issues.apache.org/jira/browse/SOLR-2891)
// *CompoundWordTokenFilter blindly adds term length to offset, but this can take things out of bounds
// wrt original text if a previous filter increases the length of the word (in this case ü -> ue)
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/TestHyphenationCompoundWordTokenFilterFactory.java b/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/TestHyphenationCompoundWordTokenFilterFactory.java
index 0039e20..2dfc63c 100644
--- a/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/TestHyphenationCompoundWordTokenFilterFactory.java
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/TestHyphenationCompoundWordTokenFilterFactory.java
@@ -47,6 +47,27 @@ public class TestHyphenationCompoundWordTokenFilterFactory extends BaseTokenStre
}
/**
+ * just tests that the two no configuration options are correctly processed
+ * tests for the functionality are part of {@link TestCompoundWordTokenFilter}
+ */
+ public void testLucene8183() throws Exception {
+ Reader reader = new StringReader("basketballkurv");
+ TokenStream stream = new MockTokenizer(MockTokenizer.WHITESPACE, false);
+ ((Tokenizer)stream).setReader(reader);
+ stream = tokenFilterFactory("HyphenationCompoundWord",
+ "hyphenator", "da_UTF8.xml",
+ "dictionary", "compoundDictionary_lucene8183.txt",
+ "onlyLongestMatch", "false",
+ "noSubMatches", "true",
+ "noOverlappingMatches", "false").create(stream);
+
+ assertTokenStreamContents(stream,
+ new String[] { "basketballkurv", "basketball", "kurv"},
+ new int[] { 1, 0, 0}
+ );
+ }
+
+ /**
* Ensure the factory works with no dictionary: using hyphenation grammar only.
* Also change the min/max subword sizes from the default. When using no dictionary,
* it's generally necessary to tweak these, or you get lots of expansions.
@@ -61,7 +82,7 @@ public class TestHyphenationCompoundWordTokenFilterFactory extends BaseTokenStre
"maxSubwordSize", "4").create(stream);
assertTokenStreamContents(stream,
- new String[] { "basketballkurv", "ba", "sket", "bal", "ball", "kurv" }
+ new String[] { "basketballkurv", "ba", "sket", "ball", "bal", "kurv" }
);
}
diff --git a/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/compoundDictionary_lucene8183.txt b/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/compoundDictionary_lucene8183.txt
new file mode 100644
index 0000000..1e62f89
--- /dev/null
+++ b/lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/compoundDictionary_lucene8183.txt
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# A set of words for testing LUCENE-8183
+basketball
+basket
+ball
+kurv
|
|
Martin Demberger (migrated from JIRA) Hi @uschindler, I have created a pull request apache/lucene-solr#2014 with the changes from Rupert Westenthaler. |
|
Uwe Schindler (@uschindler) (migrated from JIRA) Hi, |
|
Martin Demberger (migrated from JIRA) Hi Uwe, is there any way I can fasten the merge. We have a few customer which wait for this fix because our search very often falls into this pithole. |
|
Martin Demberger (migrated from JIRA) Is there any update on this? I have updated the PR so it approves to the new checks. |
|
Are there still plans to merge this? I would also be very interested in this feature. |
|
Hi @uschindler, |
|
Hi thanks @MartinDemberger , I think it should be fine to merge this into head, but I'd like to get another look by @rmuir who was one of the committers working on that TokenFilter. If this also fixes the problems with my dictionary and the configuration presented on its repository (https://github.com/uschindler/german-decompounder) I am more than happy. To be clear: Except reordering tokens there aren't any backwards compatibility issues by the new features? From what I understood it only removes useless tokens - order of tokens with same position does not matter. So actually somebody having an index that was created with the older version of that filter won't see any serious issues, just some inprecise matches may no longer be returned (because either the token is no longer in new documents of the index or the generated query no longer contains the token). So it would only return less matches, but no wrong matches. To me this looks fine. |
|
Does this also fix #4096 ? |
You're right. I did not notice that this is the dictionary one. I have just seen that this issue was linked to the other one and did not read the whole issue. |
|
Hi, Would it be possible that you maybe give a PR for the german-decompounder with an updated configuration in README.txt. Solr should not be an issue, maybe some documentation fixes there, too. |
|
Hi, I will create the ES/OS issue and try to update the README in the next days. |
The HyphenationCompoundWordTokenFilter creates overlapping tokens even if onlyLongestMatch is enabled.
Example:
Dictionary:
gesellschaft,schaftHyphenator:
de_DR.xml//from Apche OffoonlyLongestMatch: true
IMHO this includes 2 unexpected Tokens
Migrated from LUCENE-8183 by Rupert Westenthaler, 1 vote, updated Jan 14 2021
Environment:
Attachments: LUCENE-8183_20180223_rwesten.diff, LUCENE-8183_20180227_rwesten.diff, lucene-8183.zip
Linked issues:
The text was updated successfully, but these errors were encountered: