v0.28: Hebrew language support

[guimachiavelli]

As title states, Meilisearch v0.28 will have official support for Hebrew.

We need to update the "Languages" page with this information. More info required on the tokenizer and whether we need to specify it as we have done for Chinese and Japanese.

References

Issue on core: meilisearch/meilisearch#2417
Internal Meilisearch library for tokenizer management: https://github.com/meilisearch/charabia
SME: @ManyTheFish

[dichotommy]

As I work on this issue, I will also take a look at the tokenizer page, which is now somewhat out of date. These may be useful:

meilisearch/meilisearch#2375 (comment)
meilisearch/charabia#72

I will make some small changes to the tokenization page based on the content of these issues, and then request someone from @meilisearch/core-team to review.

[ManyTheFish]

Hey @dichotommy, don't hesitate to ask if you need some explanations! 😊

[dichotommy]

Hello @ManyTheFish , I know that since the tokenizer has been refactored, the tokenization page on the docs is most likely no longer accurate. For example, the image on that page is out of date since the preprocessing step has been removed, the "tokenizer" step has been renamed "segmenter", and we now offer more than just Latin and Chinese.

I'm wondering three main things:

1. Does the core team have a more up-to-date image to represent the analyzer/tokenization process? Of course, no problem if not, we will just remove it 😄
2. To get a technical understanding of the tokenizer, is there a more recent place to point users than the old specification? Does CONTRIBUTING.md fill that same role?
3. How would you change the text below, if at all:

When you add documents to a Meilisearch index, the tokenization process is handled by an abstract interface called an analyzer. The analyzer is responsible for determining the primary language of each field based on the scripts (e.g., Latin alphabet, Chinese hanzi, etc.) that are present there. Then, it applies the corresponding pipeline to each field.

We can break down the tokenization process like so:

Crawl the document(s) and determine the primary language for each field
Go back over the documents field-by-field, running the corresponding tokenization pipeline, if it exists

Thank you very much for your help 😊 ❤️ 🙏🏻

[ManyTheFish]

Hello @dichotommy!

Does the core team have a more up-to-date image to represent the analyzer/tokenization process? Of course, no problem if not, we will just remove it 😄

For now, I didn't make a schema that represents the new behavior of the tokenizer.
However, if you think that it's better with a schema, I can craft you a new one. 😄

To get a technical understanding of the tokenizer, is there a more recent place to point users than the old specification? Does CONTRIBUTING.md fill that same role?

You're right! There is no technical explanation of the tokenizer anymore, I'll see with @gmourier if the specification repository is the most suitable for a technical explanation of the tokenizer

How would you change the text below, if at all:

Something like below should be more accurate:

- When you add documents to a Meilisearch index, the tokenization process is handled by an abstract interface called an analyzer. The analyzer is responsible for determining the primary language of each field based on the scripts (e.g., Latin alphabet, Chinese hanzi, etc.) that are present there. Then, it applies the corresponding pipeline to each field.
+ When you add documents to a Meilisearch index, the tokenization process is handled by an abstract interface called a tokenizer. The tokenizer is responsible for splitting each field by Script (e.g., Latin alphabet, Chinese hanzi, etc.). Then, it applies the corresponding pipeline to each part of each field.

We can break down the tokenization process like so:

- Crawl the document(s) and determine the primary language for each field
+ Crawl the document(s) splitting each field by Script
- Go back over the documents field-by-field, running the corresponding tokenization pipeline, if it exists
+ Go back over the documents part-by-part, running the corresponding tokenization pipeline, if it exists

[guimachiavelli]

Closed by #1707