MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size

[bb]

Describe the bug

Upgrading from 0.21.1 to 0.22.0rc0, I get the following "internal error" in a status update from one of my "special" datasets. This is exactly the same dataset as from #1590 with roughly 800k docs which I'm sending all at once using meilisearch-ruby 0.15.4 and meilisearch-rails 0.2.1.

Post.ms_clear_index! rescue puts("post index was not there yet"); Post.ms_reindex!(1000000, true); Post.ms_index.get_all_update_status.last

 {"status"=>"failed",
  "updateId"=>299,
  "type"=>{"name"=>"DocumentsAddition"},
  "message"=>"internal: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size",
  "errorCode"=>"internal",
  "errorType"=>"internal_error",
  "errorLink"=>"https://docs.meilisearch.com/errors#internal",
  "duration"=>223.605,
  "enqueuedAt"=>"2021-09-06T15:22:08.518399936Z",
  "processedAt"=>"2021-09-06T16:27:48.919712335Z"},

This is the docker-compose config I'm using:

services:
  meilisearch:
    image: getmeili/meilisearch:v0.22.0rc0
    environment:
      MEILI_ENV: development
      MEILI_HTTP_ADDR: 0.0.0.0:7700
      MEILI_HTTP_PAYLOAD_SIZE_LIMIT: 2048576000
      MEILI_MASTER_KEY: secret
      RUST_BACKTRACE: full
    healthcheck:
      # todo reduce interval later
      interval: 5000s
      retries: 5
      test: wget --no-verbose --tries=1 --spider http://localhost:7700/health || exit
        1
      timeout: 3s
    ports:
    - 127.0.0.1:7700:7700/tcp
    restart: unless-stopped
    volumes:
    - ./data/meilisearch:/data.ms:rw

There's nothing helpful in the 0.22.0rc0 logs, here's a short excerpt:

meilisearch_1    | [2021-09-06T20:54:22Z INFO  actix_web::middleware::logger] 172.23.0.7:54376 "GET /indexes/Post/updates HTTP/1.1" 200 61617 "-" "Ruby" 0.693880
meilisearch_1    | [2021-09-06T20:54:50Z INFO  actix_web::middleware::logger] 172.23.0.7:54528 "POST /indexes/Post/documents HTTP/1.1" 202 16 "-" "Ruby" 9.616628
meilisearch_1    | [2021-09-06T21:03:54Z INFO  actix_web::middleware::logger] 172.23.0.7:59906 "GET /indexes/Post HTTP/1.1" 200 136 "-" "Ruby" 0.006361
meilisearch_1    | [2021-09-06T21:03:54Z INFO  actix_web::middleware::logger] 172.23.0.7:59908 "POST /indexes/Post/settings HTTP/1.1" 202 16 "-" "Ruby" 0.010274
meilisearch_1    | [2021-09-06T21:03:54Z INFO  actix_web::middleware::logger] 172.23.0.7:59910 "GET /indexes/Post/updates HTTP/1.1" 200 62134 "-" "Ruby" 0.722652
meilisearch_1    | [2021-09-06T21:04:10Z INFO  actix_web::middleware::logger] 172.23.0.7:60092 "GET /indexes/Post/updates HTTP/1.1" 200 62199 "-" "Ruby" 0.738793

Expected behavior

In Meilisearch 0.21.1 the same input result in the following update status:

Post.ms_index.get_all_update_status.last
 => {"status"=>"processed", "updateId"=>309, "type"=>{"name"=>"DocumentsAddition", "number"=>788551}, "duration"=>528.46, "enqueuedAt"=>"2021-09-06T21:20:13.695926521Z", "processedAt"=>"2021-09-06T21:29:02.162949784Z"}

Also, there are lot more log messages in <=0.21.1, like words pairs proximity (...) is too large to be saved, word "..." is too large to be saved but at least it ended with

meilisearch_1 | [2021-09-06T21:29:00Z INFO meilisearch_http::index::updates] document addition done: DocumentAdditionResult { nb_documents: 788551 }

MeiliSearch version: [e.g. v0.22.0rc0]

[curquiza]

Hello @bb

Thanks for your report, we will come back to you soon

[curquiza]

@bb this should be fixed in v0.22.0 😉
Thank you again for your report

[curquiza]

Fixed by #1686

[bb]

Thanks for your support @curquiza! I can confirm the fix is working for me.

Also, I noticed an indexing speed increase of about 15% compared to version 0.21. Congratulations, great work @ManyTheFish 👏