New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expose WordDelimiterGraphTokenFilter #23327
Conversation
This change exposes the new Lucene graph based word delimiter token filter in the analysis filters. Unlike the `word_delimiter` this token filter named `word_delimiter_graph` correctly handles multi terms expansion at query time. Closes #23104
Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM; just two minor comments. Thanks @jimczi!
import java.io.StringReader; | ||
|
||
/** | ||
* Base class to test {@link WordDelimiterGraphTokenFilterFactory} and {@link WordDelimiterGraphTokenFilterFactory} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One of these shouldn't have Graph
in it?
|
||
experimental[] | ||
|
||
Named `word_delimiter_graph`, it Splits words into subwords and performs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lowercase splits
here?
Thanks @mikemccand ! |
This change exposes the new Lucene graph based word delimiter token filter in the analysis filters. Unlike the `word_delimiter` this token filter named `word_delimiter_graph` correctly handles multi terms expansion at query time. Closes #23104
* master: (26 commits) CLI: Fix prompting for yes/no to handle console returning null (elastic#23320) Tests: Fix reproduce line for packagingTest (elastic#23365) Build: Remove extra copies of netty license (elastic#23361) [TEST] Removes timeout based wait_for_active_shards REST test (elastic#23360) [TEST] increase timeout slightly in wait_for_active_shards test to allow for index creation cluster state update to be processed before ensuring the wait times out Handle snapshot repository's missing index.latest Adding equals/hashCode to MainResponse (elastic#23352) Always restore the ThreadContext for operations delayed due to a block (elastic#23349) Add support for named xcontent parsers to high level REST client (elastic#23328) Add unit tests for ParentToChildAggregator (elastic#23305) Fix after last merge with master and apply last comments [INGEST] Lazy load the geoip databases. disable BWC tests for the highlighters, need a new 5.x build to make it work Expose WordDelimiterGraphTokenFilter (elastic#23327) Test that buildCredentials returns correct clazz (elastic#23334) Add BreakIteratorBoundaryScanner support for FVH (elastic#23248) Prioritize listing index-N blobs over index.latest in reading snapshots (elastic#23333) Test: Fix hdfs test fixture setup on windows delete and index tests can share some part of the code Remove createSampleDocument method and use the sync'ed index method ...
* Updated api gen to 5.4 and added a way to patch specification files through special *.patch.json companion files. Due to pending discusion on elastic/elasticsearch@e579629 :q! * updated x-pack spec to 5.4 * add codegen part for xpack info related APIs * Added support for Field Caps API * add support for RemoteInfo API and adds cross cluster support to IndexName * added support for SourceExists() * add skipversion, eventhough this API existed it was undocumented prior to 5.4 * expose word delimiter graph token filter as per elastic/elasticsearch#23327 * spaces=>tabs * expose num_reduce_phases as per elastic/elasticsearch#23288 * implemented XPackInfo() started on XPackUsage() * added response structure for XPackUsage() * change license date from DateTime to DateTimeOffset' * implement PR feedback on #2743 * remove explicit folder includes in csproj files
This change exposes the new Lucene graph based word delimiter token filter in the analysis filters.
Unlike the
word_delimiter
this token filter namedword_delimiter_graph
correctly handles multi terms expansion at query time.Closes #23104