Prevent DirectCandidateGenerator to reuse an unclosed analyzer #12670
Conversation
When postFilter generates a token that is identical to the input term DirectCandidateGenerator should not preFilter this token. If postFilter and preFilter are the same analyzer instance it would fail with : "TokenStream contract violation: close() call missing"
|
Hi @nomoa! You'll have to sign the CLA before I can merge this but I'll review it now. |
|
hey, can you maybe sign the CLA so we can merge it? |
|
Signed, it's weird because I already signed it last time... I will re-sign :) |
|
OK signed again with the correct email address :) |
| @@ -152,7 +152,7 @@ public void nextToken() throws IOException { | |||
|
|
|||
| if (posIncAttr.getPositionIncrement() > 0 && result.get().bytesEquals(candidate.term)) { | |||
| BytesRef term = result.toBytesRef(); | |||
| long freq = frequency(term); | |||
| long freq = internalFrequency(term); | |||
There was a problem hiding this comment.
Maybe a comment on why internalFrequency is the right thing to do here.
|
LGTM I think this is because of synonym handling maybe? I with I remember / would have documented it :( |
|
LGTM. Thanks for adding the comments. |
|
I'll do the merging if that is ok with @s1monw . |
|
++ |
Prevent DirectCandidateGenerator to reuse an unclosed analyzer
|
Thanks! |
|
Thanks @nomoa ! |
When postFilter generates a token that is identical to the input term DirectCandidateGenerator should not preFilter this token. If postFilter and preFilter are the same analyzer instance it would fail with : "TokenStream contract violation: close() call missing" This is a forward port of @nomoa's #12670
|
And I've finally merged this to master. I couldn't figure out a good way to forward port it and keep your name on it, sorry about that. I referenced you in the commit message at least. |
When postFilter generates a token that is identical to the input term
DirectCandidateGenerator should not preFilter this token. If postFilter
and preFilter are the same analyzer instance it would fail with :
"TokenStream contract violation: close() call missing"