Meilisearch key deletion

[axotion]

Hi!

I have a question regarding deleting keys. Why they are staying in auth/data.md file after all? It does not make sense to me to be honest and It raises some problem with production enviroment, when for example we are issuing always new keys to users and remove the old (because of expirations). At the end of the month we could end up with a data.md file with over N GB size.... which does not seems to be expected behaviour

Any idea why it is done in this particular way? Or maybe I'm missing something here? Will it grow indefinitely?

EDIT: I see the same is happening to tasks, they are staying in tasks/data.md folder. In my prod environment it already takes

563.2M  ./tasks

EDIT2: I see that there is an endpoint to remove old tasks - https://www.meilisearch.com/docs/reference/api/tasks#delete-tasks

[macraig]

Hi @axotion 👋

Why they are staying in auth/data.md file after all?

This has to do with how LMDB works. As long as the record is being read, it will be kept in the file.

Will it grow indefinitely?

No. Once you replace the key with a new one, it will stop being accessed and will eventually be marked as deleted. It might take a bit of time though.

You can read more about how LMDB works in our Storage docs.

[axotion]

Hi, thanks for clarification!

I just want to make sure about one thing

be marked as deleted

Does marking as deleted means that it will be eventually removed from file, and reduce size?

I read those docs

As a result, disk space occupied by LMDB (and thus by Meilisearch) tends to increase over time. It is not possible to calculate the precise maximum amount of space a Meilisearch instance can occupy.

But they still are not clear to me whenever file size will shrink when e.g I mark all keys are deleted

[macraig]

Hi again!

Does marking as deleted means that it will be eventually removed from file, and reduce size?

It won't be removed, it will be marked as reusable space that can be overwritten. Future data will overwrite the data in the file. At no point will the file reduce in size, but it won't grow indefinitely either.