Task DB is not automatically cleared and takes up a lot of space for frequent updates to indexes

[mmachatschek]

Describe the bug

I discovered an issue with our self hosted meilisearch instance that had limited resources to a 1GB disk. Our indexes are fairly small (5 indexes with 100-15.000 documents each) so updates to the index are very fast. We have very frequent updates to the indexes and are happy with the performance of both the search and processing of the task backlog.
After a few (3) weeks of running the meilisearch instance, we started to see the error message Internal error: No space left on device (os error 28) when adding a document update.
The indexes sizes where around 150MB in total on disk which should not have exhausted our disk space.
The size of data.ms/data.mdb was at 800MB. Querying the /tasks route showed 1.020.000 tasks.
After increasing the disk size, the meilisearch instance started to process the tasks again and also answered to any sort of updates to settings/documents/indexes etc.

Having an instance of meilisearch running for a very long time that has a limited amount of disk space would eventually run into this problem as well. Using the smallest server of the meilisearch cloud offering would give me 10GB of storage which would most likely raise this error after about 3 months of running the instance.

To Reproduce
Steps to reproduce the behavior:

1. Add an index
2. Add documents (doesn't need to be a lot of data)
3. Add a lot of updates to documents (which generate tasks in meilisearch)
4. observe that the data.ms/data.mdb grows

Expected behavior

1. I would expect meilisearch to automatically clean up completed tasks (or make it configurable when to clean up tasks e.g. everything older than a day that have been completed)
2. The --max-task-db-size setting should limit the task DB size to the amount specified (as it currently does) and should only break with the error message Internal error: MDB_MAP_FULL: Environment mapsize limit reached if the task DB is full with enqueued tasks. Succeeded tasks should be deleted if possible to save space and make room for new enqueued tasks

Things I tried

1. Set the --max-task-db-size -> made the problem even worse as the task db size limit was reached faster
2. Dumping meilisearch and reimport it -> takes very long with a big task DB size but the size of data.ms/data.mdb stays the same after reimporting
3. increasing the disk size -> only temporarily fixes the issue as the data.ms/data.mdb file will continue to grow when adding more updates to meilisearch
4. Recreate the meilisearch instance with indexes and clearing the data.ms -> not really a nice fix as this would need to manually login to the server and running commands to both restart meilisearch with a cleared folder and recreate the indexes

Meilisearch version: [e.g. v0.20.0]
The behaviour was discovered with >v.0.27.2 but most likely exists with all meilisearch versions that use the task/update api

[curquiza]

Hello @mmachatschek 👋

Nice to see you around, sorry it's to open an issue!
Thanks a lot for this detailed report! 😄

Most of the core team is on Holiday right now, but be sure we will investigate on it asap!
Pinging the whole @meilisearch/core-team, but more specifically @gmourier, @Kerollmops and @irevoire (since the use case could be related to #2583 in the future)

[mmachatschek]

Hi @curquiza!
I opened this issue based on a short chat with @Kerollmops. He also received a dump file if it helps for any further debugging, but I think the reproduction is fairly easy.

[curquiza]

Hello everyone following this issue.

We indeed plan to add a feature to flush old tasks in v0.30.0, see the current issue: #2751

No automation for deleting tasks are planned so far, however we are currently working on it to know if we will add it or not.
Everyone can let their opinions and feedback on the corresponding product discussion meilisearch/product#517

I close this issue then in favor of meilisearch/product#517 and #2751