Weird highlight results with non-ascii

[OrkhanAlikhanov]

Describe the bug
Meilisearch is giving weird results compared to algolia and typesense.

To be honest I don't know if the software is built to behave in this way, or the language I use is unsupported. I wanted to report this in any case. I there is a settings toggle for it to become strict, I'd like to know.

To Reproduce
Steps to reproduce the behavior:

1. Go to meillisearch dashboard
2. Search for "Müəllimlərin İşə Qəbulu - Təsviri İncəsənət" or "Təsviri İncəsənət"
3. See the weird results

Expected behavior
Should show correct results as algolia and typesense.

Screenshots
Check out the video where I explain the results. Don't forget to enable audio.

meilisearch_issue.mp4

if you want, here is youtube link https://www.youtube.com/watch?v=kumI6XbjnUA

Meilisearch version: v0.22.0, v0.25.2

[OrkhanAlikhanov]

With ascii-only it seems more correct, but I still would like to get single result in both cases

(Make sure to enable audio)

Screen.Recording.2022-02-05.at.20.15.48.mov

[Kerollmops]

Hey @OrkhanAlikhanov,

Thank you very much for your issue and detailed explanation!
This bug seems to be highly related to the default Meilisearch tokenizer, Algolia uses a Unicode segmentation algorithm, and Typesense, IIRC, uses a basic split by whitespace system. We use a custom algorithm and it seems like it is the source of the issue here. BTW we do not support Turk, IIRC.

However, we plan to improve the tokenizer we use and we will probably expose a Unicode segmentation system too. @ManyTheFish plan to work on this for future releases. We will also expose parameters to disable the remove-words-at-the-end-of-the-query for when there are not enough documents to fulfill the 20 results.

[OrkhanAlikhanov]

Hey! Thank you for the detailed answer. How can one contribute to this. Is it easy to update tokenizer function or? Any directions if I want to mess around to workaround this issue?

[Kerollmops]

Hey! You should look into our tokenizer repository, it is the crate we use in Meilisearch, more specifically in the milli crate.

However, could first create an issue and wait until next week for an answer as we plan to rewrite the tokenizer and @ManyTheFish is the one who will do this. I would prefer that he plan the rewriting with you. You can always propose a patch on the tokenizer and patch meilisearch with your fork first.

[curquiza]

I put this issue in v0.28.0 Milestones. The new tokenizer might fix this, nothing sure, we need to test once the first RC is done

[ManyTheFish]

Hello @curquiza, after some tests I'm confident that the new tokenizer will fix this issue. 😄

[curquiza]

So this issue will be fixed when #2375 will be fixed 🚀