Force japanese v1.1.0 #3588
Force japanese v1.1.0 #3588
Conversation
Uffizzi Ephemeral Environment
|
ManyTheFish
force-pushed
the
force-japanese-v1.1.0
branch
3 times, most recently
from
cdf1a01
to
8bcfbee
Compare
ManyTheFish
force-pushed
the
force-japanese-v1.1.0
branch
from
8bcfbee
to
f779548
Compare
3667: Disable autobatching of additions and deletions r=irevoire a=dureuill # Pull Request ## Related issue Fixes #3664 ## What does this PR do? - Modifies the autobatcher to not batch document additions and deletions, as a workaround to the DB corruption in #3664 Co-authored-by: Louis Dureuil <louis@meilisearch.com>
3659: stops receiving tasks once the task queue is full r=Kerollmops a=irevoire Give 20GiB to the task queue + once 50% of the task queue is used, it blocks itself and only receives task deletion requests to ensure we never get in a state where we can’t do anything. Also, create a new error message when we reach this case: ``` Meilisearch cannot receive write operations because the size limit of the tasks database has been reached. Please delete tasks to continue performing write operations. ``` Co-authored-by: Tamo <tamo@meilisearch.com>
…ter deleting tasks
Co-authored-by: Tamo <tamo@meilisearch.com>
3673: Handle the task queue being full r=irevoire a=dureuill # Pull Request ## Related issue Fixes a remaining issue with #3659 where it was not always possible to send tasks back even after deleting some tasks when prompted. ## Tests - see integration test - also manually tested with a 1MiB task queue. Was not possible to become unblocked before this PR, is now possible. ## What does this PR do? - Use the `non_free_pages_size` method to compute the space occupied by the task db instead of the `real_disk_size` which is not always affected by task deletion. - Expand the test so that it adds a task after the deletion. The test now fails before this PR and succeeds after this PR. Co-authored-by: Louis Dureuil <louis@meilisearch.com>
|
It seems that a panic occurs when trying to index documents containing "ッー". This issue has occurred in both
panic message Related: PSeitz/wana_kana_rust#13 (Reported by @kounoike) |
|
@miiton, Thank you for your report, I'll fix it ASAP! |
|
Hello @miiton, A new prototype up-to-date with v1.1.1 has been released hot fixing the issue you pointed out before:
I updated the PR description in consequence. |
|
I thought it was a nice feature to be able to search for katakana words in hiragana (and vice versa), but this fix has disabled that once and for all, correct? prototype-japanese-1 curl ... -X POST localhost:7700/indexes/hogehoge/documents -d '{"id":1,"name":"パイナップル"}'
# {"taskUid":0,"indexUid":"hogehoge","status":"enqueued","type":"documentAdditionOrUpdate","enqueuedAt":"2023-05-04T23:28:15.037778125Z"}
curl ... -X POST localhost:7700/indexes/hogehoge/search -d '{"q":"パイナップル"}'
# {"hits":[{"id":1,"name":"パイナップル"}],"query":"パイナップル","processingTimeMs":2,"limit":20,"offset":0,"estimatedTotalHits":1}
curl ... -X POST localhost:7700/indexes/hogehoge/search -d '{"q":"ぱいなっぷる"}'
# {"hits":[{"id":1,"name":"パイナップル"}],"query":"ぱいなっぷる","processingTimeMs":0,"limit":20,"offset":0,"estimatedTotalHits":1} -- hitprototype-japanese-2 curl ... -X POST localhost:7700/indexes/hogehoge/documents -d '{"id":1,"name":"パイナップル"}'
# {"taskUid":0,"indexUid":"hogehoge","status":"enqueued","type":"documentAdditionOrUpdate","enqueuedAt":"2023-05-04T23:28:15.037778125Z"}
curl ... -X POST localhost:7700/indexes/hogehoge/search -d '{"q":"ぱいなっぷる"}'
# {"hits":[],"query":"ぱいなっぷる","processingTimeMs":0,"limit":20,"offset":0,"estimatedTotalHits":0} -- not hit |
|
Hello @miiton, |
Pull Request
This PR deactivates Chinese tokenization forcing Japanese tokenization to be used by Meilisearch.
It's a hotfix provided to Japanese users allowing them to use Meilisearch without having language detection issues before a final fix is released.
How to use this prototype?
Docker images
Meilisearch v1.1.1 (latest):
Meilisearch v1.1.0-rc.3:
Meilisearch v1.1.0-rc.1:
Related issues and discussions