An Emoji capable tokenizer

[damienalexandre]

We need a specific tokenizer to handle emoji properly in Elasticsearch, here is the specification.

Emoji Tokenizer

Should be based on the standard tokenizer to keep the grammar logic and Unicode Text Segmentation implementation.

But add support for symbols / emoji as specified in tr51.

Test cases

Input	Tokens
Get some donut's.	Get some donut's
Get some 🍩's.	Get some 🍩's
"🍩"	🍩

[damienalexandre]

Currently working on a ICU 58 based tokenizer with additional custom rules to consider emoji & their combinaisons as word/token.
Elasticsearch's ICU plugin is out of date (ICU 54), Lucene ICU too (ICU 56), so I will have to ship my own ICU with the plugin, my only concern is about compatibility...

😋

[gibrown]

@damienalexandre did you manage to build an ICU tokenizer that supports emoji tokenization by any chance?

[damienalexandre]

Yes! The plugin documentation is here: https://github.com/jolicode/emoji-search/tree/master/esplugin

[gibrown]

Looks really cool! Thanks! Adding to our project list to try it out.