Analysis plugin for ElasticSearch providing capability for processing inline annotations in documents.
Inline annotations are considered to be simple semantic informations inlined in source text, that are removed from the indexed text and injected as synonyms at positions of the words they are related to.
This plugin provides analyzer
AnnotationAnalyzer as well as filter
AnnotationAnalyzer is composed of
InlineAnnotationFilter (with default settings).
More sophisticated analyzers (equivalent to StandardAnalyzer or SnowballAnalyzer)
can be configure via configuration file elasticsearch.yml or web API.
Let's say we have this documents
"Mozart[artist] was born[lifeEvent] in Salzburg[city;Austria]"
If we parse this with StandardAnalyzer equivalent with annotation analysis added to it we get these tokens - some are omitted due to used StopFilter.
| [austria] [artist] | | [lifeevent] | | [city] mozart | | born | | salzburg
If we use StandardAnalyzer the result would be
mozart | artist | | born | lifeevent | | salzburg | city | austria
This plugin follows conventions for elasticsearch plugins, thus can be installed in a standard manner - see http://www.elasticsearch.org/guide/reference/modules/plugins/
Using this plugin
To use those custom analyzers/filters you need to either modify
configuration file - see http://www.elasticsearch.org/guide/reference/index-modules/analysis/ or specify
index mapping via elasticsearch API.
The following example configuration contains definitions for analyzers based on behaviour of StandardAnalyzer and SnowballAnalyzer.
Please note that standard_annotation and snowball_annotation analyzers use standard tokenizer, which removes all non-alphanumeric characters and thus makes it impossible to process inline annotations marked with [,],; (which are used in default behaviour of InlineAnnotationFilter).
For this purpose we need to use mapping char filter, which remaps those special characters to their equivalent, which will be accepted by standard tokenizer as part of the token.
index : analysis : char_filter : annotation_remap : type : mapping mappings : ["[=>__annotation_start__", "]=>__annotation_end__",";=>__annotation_delimiter__"] analyzer : standard_annotation : type : custom tokenizer : standard char_filter : annotation_remap filter : [standard, lowercase, annotation_filter, stop] snowball_annotation : type : custom tokenizer : standard char_filter : annotation_remap filter : [standard, lowercase, annotation_filter, stop, snowball] filter : annotation_filter : type : annotation_filter start : __annotation_start__ end : __annotation_end__ delimiter : __annotation_delimiter__
To test the analyzer you can query the following http://localhost:9200/test/_analyze?analyzer=annotation&text="Mozart[city;Salzburg]"
Another thing to keep in mind is that you can't use word-delimiting characters inside annotations. The whole string would be treated as two tokens which would result in unexpected behaviour.
The InlineAnnotationFilter can be slightly customized.
List of supported options
start- start delimiter for inline annotation
end- end delimiter for inline annotation
prefix- string to be prepended to synonym, that is created from inline annotation
suffix- string to be apended to synonym, that is created from inline annotation
token-type- token type of synonym
delimiter- delimiter for multiple inline annotations
Example providing default values
index : analysis : analyzer : annotation : type : annotation start : [ end : ] prefix : [ suffix : ] token-type: synonym delimiter : ;
This plugin was successfuly tested on elasticsearch version 0.90.2