Load a user-defined dictionary

[gmourier]

Why?

Some Languages are not always space separated (Chinese, Japanese, Korean...) forcing Meilisearch to rely on common word dictionaries to segment a text into words, however, this strategy works only to segment common words and struggles when a text contains domain-specific words, technical words, or People Nouns.

Because it is impossible to have an exhaustive list of words of all the possible domains or a complete list of Nouns, Meilisearch should allow the user to add their own list of words specialized for their use case.

Relative discussions/issues

#3590
#2289
#1879
#135
532#discussioncomment-5995300
#468
#160 (comment)
meilisearch/meilisearch#2879

Relative discord messages

https://discord.com/channels/1006923006964154428/1120407180823232672/1120407180823232672

[macraig]

Hello everyone 👋

We just released a 🧪 prototype that allows customizing tokenization and we'd love your feedback.

How to get the prototype?

Using docker, use the following command:

docker run -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:prototype-tokenizer-customization-2

From source, compile Meilisearch on the prototype-tokenizer-customization-2 tag

How to use the prototype?

You can find all the details in the PR.

⚠️ We do NOT recommend using this prototype in production. This is for test purposes only.

Feedback and bug reporting when using this prototype are encouraged! Thanks in advance for your involvement. It means a lot to us ❤️

[macraig]

Hey everyone! Quick update here, a new version of the prototype has been released:

$ docker pull getmeili/meilisearch:prototype-tokenizer-customization-3

This new version fixes an issue our team found on the separator-tokens routes.

[macraig]

Hello everyone 👋

We have just released the first RC (release candidate) of Meilisearch containing this new feature!

You can test it by using:

• The release assets
• The Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.4.0-rc.0

You are welcome to leave your feedback in this discussion.

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 Official and stable release containing this change will be available on September 25th, 2023

⚠️ RC (release candidates) are not recommended for production

[macraig]

Hey folks 👋

v1.4.0 has been released! 🦓 You can now expand the default dictionaries and improve accuracy when using domain-specific terms. ✨

Note:

📚 https://www.meilisearch.com/docs/reference/api/settings#dictionary