MDB_KEYEXIST: Key/data pair already exists.

[Ardnozz]

Describe the bug
I am experiencing a bug when I cannot post any request to meilisearch database because of the addition, replacement, and deletion of the only one lasting document in an index. The log of the middleware inside meilisearch is still registering them, but it is marking them as failed with unreasonable error.

To Reproduce
Steps to reproduce the behavior:

1. Create an empty index and set the primary key for "uuid"
2. Add one document to the index
3. Replace the document with another value
4. Delete the document (so there is no document inside the index)
5. Add a new document and instead of its creation you will see a failed task (even though the uuid is different, this should not be the cause)

Expected behavior
The database should add new documents without any errors and failed tasks.

Screenshots
Failed task

Meilisearch version:
v1.5.0

Additional context
I am using the docker version of meilisearch built on WSL and for posting the request to the database I use an official meilisearch-js API.

[dureuill]

Hello @Ardnozz 👋

Thank you very for the very actionable report ✨

It looks like you encountered an error with the engine, I'm sorry that this is happening 😞

Unfortunately, even by following your detailed steps, I cannot reproduce the issue you're facing, be it on v1.5.0 or on v1.6.0-rc.4.

Here is the list of curl commands I executed:

# 1. Create an empty index and set the primary key for "uuid"
curl http://localhost:7700/indexes -H 'Content-Type: application/json' --data-binary '{"uid": "lecturers", "primaryKey": "uuid"}'
# 2. Add one document to the index
curl http://localhost:7700/indexes/lecturers/documents -H 'Content-Type: application/json' --data-binary '[{"uuid": "510fde4e-a07a-4676-8b07-680f32f3ecd7", "title": "initial document"}]'
# 3. Replace the document with another value
curl http://localhost:7700/indexes/lecturers/documents -H 'Content-Type: application/json' --data-binary '[{"uuid": "510fde4e-a07a-4676-8b07-680f32f3ecd7", "title": "replaced document"}]'
# 4. Delete the document (so there is no document inside the index)
curl -X DELETE http://localhost:7700/indexes/lecturers/documents/510fde4e-a07a-4676-8b07-680f32f3ecd7
# 5. Add a new document
curl http://localhost:7700/indexes/lecturers/documents -H 'Content-Type: application/json' --data-binary '[{"uuid": "d50708bb-2893-474f-90ee-58d3dba39900", "title": "after delete"}]'

With the above, all the tasks are succeeding as expected.

Could you perhaps check that you do reproduce the issue using the above reproducer? If not, is there any step I misinterpreted? Can you modify the script above to trigger the issue?

I really need to be able to reproduce so that I can see what is happening and hopefully find a fix 🙁

Thank you for your cooperation, and again, for your great report ❤️

[Ardnozz]

Hello @dureuill

I've tried to reproduce the bug with your curl commands and everything works perfectly. So I checked my system again and I found out that the problem is in updateFilterableAttributes action. Here is a list of curl commands that worked for me. The only difference is the API key, but I believe that it cannot do that much different.

# 1. Create an empty index and set the primary key to "uuid"
curl --location 'http://localhost:7700/indexes' --header 'Content-Type: application/json' --header 'Authorization: Bearer OURSUPERSECRETKEY' --data '{"uid": "lecturers", "primaryKey": "uuid"}'
#2. Update filterable attributes with "uuid"
curl --location --request PUT 'http://localhost:7700/indexes/lecturers/settings/filterable-attributes' --header 'Content-Type: application/json' --header 'Authorization: Bearer OURSUPERSECRETKEY' --data '["uuid"]'
#3. Add one document to the index
curl --location 'http://localhost:7700/indexes/lecturers/documents' --header 'Content-Type: application/json' --header 'Authorization: Bearer OURSUPERSECRETKEY' --data '[{"uuid": "2e7e7f4c-9402-4abf-9f5d-eb9157ee79a5", "title": "Initial title"}]'
#4. Replace the document in the index
curl --location 'http://localhost:7700/indexes/lecturers/documents' --header 'Content-Type: application/json' --header 'Authorization: Bearer OURSUPERSECRETKEY' --data '[{"uuid": "2e7e7f4c-9402-4abf-9f5d-eb9157ee79a5", "title": "Next title"}]'
#5. Delete the only document
curl --location --request DELETE 'http://localhost:7700/indexes/lecturers/documents/2e7e7f4c-9402-4abf-9f5d-eb9157ee79a5' --header 'Authorization: Bearer OURSUPERSECRETKEY'
#6. Try adding new document with completly new "uuid" and "title" property.
curl --location 'http://localhost:7700/indexes/lecturers/documents' --header 'Content-Type: application/json' --header 'Authorization: Bearer OURSUPERSECRETKEY' --data '[{"uuid": "510fde4e-a07a-4676-8b07-680f32f3ecd7", "title": "Initial title2"}]'

Hope this will work (or don't work 😅) as expected. (The curls are autogenerated by Postman)
Ardno

[dureuill]

Hello @Ardnozz

Unfortunately, I'm still unable to reproduce the issue 🥺

I ran the reproducer you provided as a bash script, but all my tasks still succeeded.
I retried multiple times on fresh DBs by interspacing a task on an unrelated index to check if it could be autobatching shenanigans, but the behavior is stable (success) regardless of the task batching behavior.

Can you confirm that running the reproducer from your message on a fresh Meilisearch DB results in exposing the issue?

If so, maybe something with your setup is causing the issue 🤔

The only peculiar thing I can see is that you're running WSL (also, you're using docker and I'm running a local v1.5 version on Mac).

1. Are you running WSL 1 or WSL 2? (if you don't know, that's probably WSL2)
2. if running WSL2, which distribution are you using?
3. Do you have the possibility and the time to try the reproducer in a different environment?

If not, I'll try to reproduce your setup on my side.

Thanks a lot for your reactivity and help in getting to the bottom of this 💯

[Ardnozz]

You're supposing right. I'm using WSL with Docker.

# wsl --version
Version WSL: 1.2.5.0
Version core: 5.15.90.1
Version WSLg: 1.0.51
Version MSRDC: 1.2.3770
Version Direct3D: 1.608.2-61064218
Version DXCore: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Version Windows: 10.0.19045.3803

# wsl --status
Default distro: docker-desktop-data
Default version: 2

We've tried the same version on Linux and it works as expected. Maybe if you wouldn't be able to reproduce it, I can switch to a new version and check if that works.

[Ardnozz]

I've tried upgrading to the latest version v1.6.0-rc.6 and built it from Dockerfile for me and downloaded the binary located in /bin/meilisearch so it can be as a part of another Docker image. The problem persists.

[dureuill]

Hello again,

Could you please clarify what you mean by:

We've tried the same version on Linux and it works as expected.

it is a bit unclear to me.

Does that mean that:

1. The issue persists on Linux?
2. The issue doesn't reproduce on Linux?

If it is (2), then it means it is a WSL specific issue. I checked with the team, and we're not officially supporting WSL at the moment, so trying to reproduce and fix on WSL will be low priority. I apologize for not telling you earlier, it wasn't clear to me. We'd be open to external contributions to fix these (but the issue might be related to how WSL implements its file system)

If it is (1), then we need further investigation on the docker image.

[Ardnozz]

Hello,
I will try concluding everything mentioned before just in case of misunderstanding, because, as you've mentioned before, it is probably a problem of WSL, that you're not currently supporting at the present moment, so it could be probably closed without resolving.

I wanted to use the meilisearch database for a project that needs to be transformed into one Docker image (so I can't use your image because I am not allowed to use Docker compose). Because I am a Windows user, I use Docker Desktop and Docker Engine specified for Windows OS. That means the Docker Desktop software is running on WSL, which, as you've mentioned, you're not supporting, even though it's just a tool for running Docker on Windows OS.

I tried to install your built versions (available here) into the docker image based on Alpine Linux (by installing the binary via the curl command), but I haven't managed to run it successfully. So I duplicated your repository and built the Dockerfile by myself. Then I downloaded the binary called meilisearch from /bin and tried pasting it into another docker image. Then after I installed the dependency (libgcc), ran it and it worked fine.

The only error that has occurred was the one that I described above. But it could be mitigated by the system that I am developing. The problem is probably either in building the version on the Docker engine that is running (by default) on WSL or in running the meilisearch on the Docker engine on Windows OS. Finally, I wanted to add that I appreciate your work and believe that someday the meilisearch database will be also available for these kinds of systems.

Thank you for your help.

[balach]

Unfortunately, I ran into the same issue just now - using Meilisearch Cloud with 1.6.0, with using firestore with the official firebase extension..

Important: We did indeed delete all documents in our index and then repopulated our databse because we were going live today. And since then the index does not update and throws the error.

The full error:
{
"indexUid": "propertiesIndex",
"status": "failed",
"type": "documentAdditionOrUpdate",
"uid": 3934,
"details": {
"receivedDocuments": 1,
"indexedDocuments": 0
},
"canceledBy": null,
"error": {
"message": "internal: MDB_KEYEXIST: Key/data pair already exists.",
"code": "internal",
"type": "internal",
"link": "https://docs.meilisearch.com/errors#internal"
},
"duration": "PT0.080720649S",
"startedAt": "2024-01-18T19:42:54.776Z",
"enqueuedAt": "2024-01-18T19:42:54.773Z",
"finishedAt": "2024-01-18T19:42:54.857Z"
}

[queer]

I ran into this today as well, using the 1.6.0-1 deb package on an Ubuntu server (no WSL or similar). Exact same error that @balach reported. Deleting the entire index and recreating it fixed the problem.

[dureuill]

Hello,

Given the number of reports coming from different platforms, it is clear that this is not an issue specific to WSL.

We discussed the issue with the team, and although we couldn't reproduce the issue yet, we have an idea what the source cause could be

Anyone encountering the issue, could you print out the settings of the misbehaving index, and one or two representative documents for that index? That would help with reproducing the issue.

I'm glad that a workaround of deleting the entire index and creating it anew workarounds the issue, thanks for letting me (and others in the issue) know, @queer

[mech]

I also encounter this issues today with "MDB_KEYEXIST: key/data pair already exists"

I am using macOS Monterey, Docker Desktop v4.27.1 with Meilisearch v1.6.1

Doing a clear_index! and reindex! again fix the issues. But I am now afraid that it will suddenly appear in production.

[Kerollmops]

Hey @mech 👋

Thank you for the report! However, can you help us reproduce the issue? Can you share your settings? The documents you were sending and the way you were sending them (batch size)?

Have a nice day 🌻

[queer]

I ran into this again yesterday (still on 1.6.0) and got some more details that may help:

• Documents are ingested in batches of 500~2500 at a time, ~540k total documents.
• Documents are often replacing existing documents, rather than inserting new documents.
• All documents have the exact same fields.
• When checking stats (GET /stats), I see:

  {
    "databaseSize": 6392389632,
    "lastUpdate": "2024-02-14T22:27:15.981578502Z",
    "indexes": {
      "products": {
        "numberOfDocuments": 0,
        "isIndexing": false,
        "fieldDistribution": {
          "_geo": 540925,
          "brand": 540925,
          // Continues for all 19 fields that are present
        }
      }
    }
  }

  Note that numberOfDocuments is 0, but fieldDistribution shows the expected numbers.
• This seems to only happen when updating documents, never on the initial insert of documents into the index.

Most recent failed tasks:

{
  "results": [
    {
      "uid": 23708,
      "indexUid": "products",
      "status": "failed",
      "type": "documentAdditionOrUpdate",
      "canceledBy": null,
      "details": {
        "receivedDocuments": 952,
        "indexedDocuments": 0
      },
      "error": {
        "message": "internal: MDB_KEYEXIST: Key/data pair already exists.",
        "code": "internal",
        "type": "internal",
        "link": "https://docs.meilisearch.com/errors#internal"
      },
      "duration": "PT0.146758390S",
      "enqueuedAt": "2024-02-15T13:30:42.710355876Z",
      "startedAt": "2024-02-15T13:30:42.711643631Z",
      "finishedAt": "2024-02-15T13:30:42.858402021Z"
    },
    {
      "uid": 23707,
      "indexUid": "products",
      "status": "failed",
      "type": "documentAdditionOrUpdate",
      "canceledBy": null,
      "details": {
        "receivedDocuments": 1349,
        "indexedDocuments": 0
      },
      "error": {
        "message": "internal: MDB_KEYEXIST: Key/data pair already exists.",
        "code": "internal",
        "type": "internal",
        "link": "https://docs.meilisearch.com/errors#internal"
      },
      "duration": "PT0.200975582S",
      "enqueuedAt": "2024-02-15T13:30:37.094515856Z",
      "startedAt": "2024-02-15T13:30:37.095765624Z",
      "finishedAt": "2024-02-15T13:30:37.296741206Z"
    },
    {
      "uid": 23706,
      "indexUid": "products",
      "status": "failed",
      "type": "documentAdditionOrUpdate",
      "canceledBy": null,
      "details": {
        "receivedDocuments": 746,
        "indexedDocuments": 0
      },
      "error": {
        "message": "internal: MDB_KEYEXIST: Key/data pair already exists.",
        "code": "internal",
        "type": "internal",
        "link": "https://docs.meilisearch.com/errors#internal"
      },
      "duration": "PT0.204137952S",
      "enqueuedAt": "2024-02-15T13:30:32.629005791Z",
      "startedAt": "2024-02-15T13:30:32.630208395Z",
      "finishedAt": "2024-02-15T13:30:32.834346347Z"
    },
    {
      "uid": 23705,
      "indexUid": "products",
      "status": "failed",
      "type": "documentAdditionOrUpdate",
      "canceledBy": null,
      "details": {
        "receivedDocuments": 540,
        "indexedDocuments": 0
      },
      "error": {
        "message": "internal: MDB_KEYEXIST: Key/data pair already exists.",
        "code": "internal",
        "type": "internal",
        "link": "https://docs.meilisearch.com/errors#internal"
      },
      "duration": "PT0.126139948S",
      "enqueuedAt": "2024-02-15T13:30:32.01057069Z",
      "startedAt": "2024-02-15T13:30:32.011849848Z",
      "finishedAt": "2024-02-15T13:30:32.137989796Z"
    }
  ],
  "total": 23709,
  "limit": 4,
  "from": 23708,
  "next": 23704
}

Index settings:

{
  "displayedAttributes": [
    "*"
  ],
  "searchableAttributes": [
    "*"
  ],
  "filterableAttributes": [
    "_geo",
    "brand",
    // Continues for all 13 filterable fields
  ],
  "sortableAttributes": [],
  "rankingRules": [
    "words",
    "typo",
    "proximity",
    "attribute",
    "sort",
    "exactness"
  ],
  "stopWords": [],
  "nonSeparatorTokens": [],
  "separatorTokens": [],
  "dictionary": [],
  "synonyms": {},
  "distinctAttribute": null,
  "proximityPrecision": "byWord",
  "typoTolerance": {
    "enabled": true,
    "minWordSizeForTypos": {
      "oneTypo": 5,
      "twoTypos": 9
    },
    "disableOnWords": [],
    "disableOnAttributes": []
  },
  "faceting": {
    "maxValuesPerFacet": 100,
    "sortFacetValuesBy": {
      "*": "alpha"
    }
  },
  "pagination": {
    "maxTotalHits": 1000
  }
}

[dureuill]

Hello all of you 👋

We recently released v1.7.0-rc.1, a pre-release version that features changes susceptible of fixing this issue.

Since we are not reproducing ourselves, could some of you who experience the issue take the time to test the new version and report if the issue still reproduces? 🙏

We would love that ❤️

⚠️ Please do not use a pre-release version in a production environment, this is for testing purposes only. The final v1.7.0 version is stated for release on March, 11th

[nlgtuankiet]

Hi @Ardnozz did you try out the v1.7.0-rc.1 version to see if this issue is fixed?

[queer]

This issue stopped reproducing on v1.6.2 for me at some point; unfortunately I never managed to pinpoint the specific problematic document.

[Kerollmops]

Hey @queer, if you are unable to reproduce on v1.6.2, it's because we fixed it in this version by disabling some parts of the engine. However, we perfectly fixed it in v1.7, and indexing is faster on this one too.