Enhance language detection

[ManyTheFish]

Summary

Some completely unrelated Languages can share the same characters, in Meilisearch we detect the Languages using whatlang, which works well on large texts but fails on small search queries leading to a bad segmentation and normalization of the query.

This PR now stores the Languages detected during the indexing in order to reduce the Languages list that can be detected during the search.

Detail

• Create a 19th database mapping the scripts and the Languages detected with the documents where the Language is detected
• Fill the newly created database during indexing
• Create an allow-list with this database and pass it to Charabia
• Add a test ensuring that a Japanese request containing kanjis only is detected as Japanese and not Chinese

Related issues

Fixes #2403
Fixes #3513

[irevoire]

Super nice. Could you add one little integration test on meilisearch just to ensure we don't disable the feature unexpectedly please?

This test converted for meilisearch should do it;

    fn store_detected_script_and_language_per_document_during_indexing() {
        use charabia::{Language, Script};
        let index = TempIndex::new();
        index
            .add_documents(documents!([
                { "id": 1, "title": "The quick (\"brown\") fox can't jump 32.3 feet, right? Brr, it's 29.3°F!" },
                { "id": 2, "title": "人人生而自由﹐在尊嚴和權利上一律平等。他們賦有理性和良心﹐並應以兄弟關係的精神互相對待。" },
                { "id": 3, "title": "הַשּׁוּעָל הַמָּהִיר (״הַחוּם״) לֹא יָכוֹל לִקְפֹּץ 9.94 מֶטְרִים, נָכוֹן? ברר, 1.5°C- בַּחוּץ!" },
                { "id": 4, "title": "関西国際空港限定トートバッグ すもももももももものうち" },
                { "id": 5, "title": "ภาษาไทยง่ายนิดเดียว" },
                { "id": 6, "title": "The quick 在尊嚴和權利上一律平等。" },
            ]))
            .unwrap();

        let rtxn = index.read_txn().unwrap();
        let key_jpn = (Script::Cj, Language::Jpn);
        let key_cmn = (Script::Cj, Language::Cmn);
        let cj_jpn_docs = index.script_language_documents_ids(&rtxn, &key_jpn).unwrap().unwrap();
        let cj_cmn_docs = index.script_language_documents_ids(&rtxn, &key_cmn).unwrap().unwrap();
        let expected_cj_jpn_docids = [3].iter().collect();
        assert_eq!(cj_jpn_docs, expected_cj_jpn_docids);
        let expected_cj_cmn_docids = [1, 5].iter().collect();
        assert_eq!(cj_cmn_docs, expected_cj_cmn_docids);
    }

[irevoire]

Perfect, thanks! 🐚
bors merge

[irevoire]

Arf, there is a conflict @ManyTheFish

[bors]

Canceled.

[github-actions]

Uffizzi Preview Environment Deploying

☁️ https://app.uffizzi.com/github.com/meilisearch/meilisearch/pull/3347

⚙️ Updating now by workflow run 4231346523.

The meilisearch preview environment contains a web terminal from where you can run the
meilisearch command. You should be able to access this instance of meilisearch running in
the preview from the link Meilisearch Endpoint link given below.

Web Terminal Endpoint :
Meilisearch Endpoint : /meilisearch

[irevoire]

Thanks! 📦
bors merge

[bors]

Build succeeded:

• Run Clippy
• Run Rustfmt
• Run tests in debug
• Tests on macos-12
• Tests on ubuntu-18.04
• Tests on windows-2022