[v1.1.0-rc.0] Japanese documents cannot be searched properly with Kanji-only documents

[miiton]

Describe the bug

I was testing the search by adding Japanese documents, but when I added a Kanji-only document, past Japanese documents were no longer caught in the search.

To Reproduce

summary

1. add documents [{id: 1, name: "東京スカパラダイスオーケストラ"}] -> search with 東京 : expected
2. add documents [{id: 2, name: "東京バナナ"}, {id:3, name: "東京 ポテチ"}] -> search with 東京 : expected
3. add documents [{id: 4, name: "東京特許許可局"}] -> search with 東京 : not expected

launch

docker run --rm -p 7700:7700 -e MEILI_MASTER_KEY=foobar getmeili/meilisearch:v1.1.0-rc.0

processing

# create an index
curl -H "Authorization: Bearer foobar" -H "Content-Type: application/json" -X POST localhost:7700/indexes -d '{"uid":"hogehoge","primaryKey":"id"}'

# add a Japanese document
curl -H "Authorization: Bearer foobar" -H "Content-Type: application/json" -X POST localhost:7700/indexes/hogehoge/documents -d '{"id":1,"name":"東京スカパラダイスオーケストラ"}'

# search and hit 1 document ( correct )
curl -H "Authorization: Bearer foobar" -H "Content-Type: application/json" -X POST localhost:7700/indexes/hogehoge/search -d '{"q":"東京"}'

{"hits":[{"id":1,"name":"東京スカパラダイスオーケストラ"}],"query":"東京","processingTimeMs":0,"limit":20,"offset":0,"estimatedTotalHits":1}

# add more documents
curl -H "Authorization: Bearer foobar" -H "Content-Type: application/json" -X POST localhost:7700/indexes/hogehoge/documents -d '{"id":2,"name":"東京バナナ"}'
curl -H "Authorization: Bearer foobar" -H "Content-Type: application/json" -X POST localhost:7700/indexes/hogehoge/documents -d '{"id":3,"name":"東京 ポテチ"}'

# search and hit 3 documents ( correct ) 
curl -H "Authorization: Bearer foobar" -H "Content-Type: application/json" -X POST localhost:7700/indexes/hogehoge/search -d '{"q":"東京"}'

{"hits":[{"id":1,"name":"東京スカパラダイスオーケストラ"},{"id":2,"name":"東京バナナ"},{"id":3,"name":"東京 ポテチ"}],"query":"東京","processingTimeMs":0,"limit":20,"offset":0,"estimatedTotalHits":3}

# add a Kanji only document
curl -H "Authorization: Bearer foobar" -H "Content-Type: application/json" -X POST localhost:7700/indexes/hogehoge/documents -d '{"id":4,"name":"東京特許許可局"}'

# search and hit only last document ( not expected )
curl -H "Authorization: Bearer foobar" -H "Content-Type: application/json" -X POST localhost:7700/indexes/hogehoge/search -d '{"q":"東京"}'

{"hits":[{"id":4,"name":"東京特許許可局"}],"query":"東京","processingTimeMs":0,"limit":20,"offset":0,"estimatedTotalHits":1}

# add a document again
curl -H "Authorization: Bearer foobar" -H "Content-Type: application/json" -X POST localhost:7700/indexes/hogehoge/documents -d '{"id":5,"name":"東京スカイツリー"}'

# search but hit only 1 document ( not expected )
curl -H "Authorization: Bearer foobar" -H "Content-Type: application/json" -X POST localhost:7700/indexes/hogehoge/search -d '{"q":"東京"}'

{"hits":[{"id":4,"name":"東京特許許可局"}],"query":"東京","processingTimeMs":0,"limit":20,"offset":0,"estimatedTotalHits":1}

Expected behavior

I expected all documents could searched by 東京

Meilisearch version: v1.1.0-rc.0

Additional context

Ref #2403 #3347

[ManyTheFish]

Hello @miiton, thank you for having tried this release candidate.

Your case is a hard one, I have a fix in mind but it's a partial one:
the documents that were searchable will continue being searchable, however, I have an issue to make '{"id":4,"name":"東京特許許可局"}' searchable 😬

I will investigate!

[ManyTheFish]

Hello @miiton,
I've tried to enhance the Language detection for Japanese support, I released a prototype on Docker, however, this prototype partially fixes the issue, see below the PR explaining the limits of this enhancement and how to try it:
#3569

Please share and try this prototype and make me some feedback!

Thanks!

[miiton]

You're so fast!

As you wrote in your PR, adding "Japanese sentences(= includes Hiragana or Katakana)" to the desc gave me the results I was expecting. So I understood that one of the key points of DB creation is that creating a DB with only words will not give the expected results.

BEFORE	AFTER

[ManyTheFish]

@miiton,
thank you for your feedback,
I will set the PR status to "ready for review" in order to merge it for v1.1.0-rc.1,
If you see some other cases where Meilisearch is wrong, don't hesitate to share them,
I think the next enhancement to do is to contribute to Whatlang as you suggested last year:
greyblake/whatlang-rs#122

[curquiza]

Fixed by #3569
Will be released in rc1 today

[miiton]

I tried to test a bit more.
It seems that the search function works well if there are many "amounts of text" in the document due to the changes merged into v1.1.0-rc1, but it doesn't work well when there are few.

When searching for 東京 after importing data_with_shortdesc.csv , only "Kanji-only words" are hit.
When searching for 東京 after importing data_with_longdesc.csv , I get roughly the desired results (I don't understand why only テレビ東京 doesn't hit).

Import command:

curl -X POST localhost:7700/indexes/:index_uid/documents --data-binary @data_with_shortdesc.csv
curl -X POST localhost:7700/indexes/:index_uid/documents --data-binary @data_with_longdesc.csv

At any rate, if I want to search in Japanese for sure, I need to put a long dummy Japanese string in each document, like data_with_longdesc.csv, otherwise it will not work well and I waste DB space in that case.

I think it would be easier to solve this problem if we could enhance the test cases in Japanese. Is there anything I can do to help?

data_with_shortdesc.csv
data_with_longdesc.csv