Huge memory usage after some time

[shyim]

Describe the bug
Uses after some time huge amount of RAM 58GB

To Reproduce
I don't know what really does that

I have a Job that deletes every hour the index and indexes all entries.
Script: https://github.com/FriendsOfShopware/packages/blob/live/src/Command/PackageIndexerCommand.php#L31-L70
We talk about only one index with 6898 entries with less fields

/var/www/html # curl mellisearch:7700/indexes
[{"name":"packages","uid":"packages","createdAt":"2021-02-04T08:12:42.654227471Z","updatedAt":"2021-02-04T08:12:52.724518077Z","primaryKey":"id"}]

/var/www/html # curl mellisearch:7700/indexes/packages
{"name":"packages","uid":"packages","createdAt":"2021-02-04T08:12:42.654227471Z","updatedAt":"2021-02-04T08:12:52.724518077Z","primaryKey":"id"}


/var/www/html # curl mellisearch:7700/indexes/packages/search
{"hits":[Removed the hits too long],"offset":0,"limit":20,"nbHits":6898,"exhaustiveNbHits":false,"processingTimeMs":0,"query":""}

Expected behavior
It should not take so much memory.

Screenshots

Server (please complete the following information):

• OS: Debian 10
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
I am running it in Docker

[Kerollmops]

Hey @shyim,

We are aware that MeiliSearch currently uses a lot of memory, we are working on a new indexing system that is much more efficient. Can you provide us the MeiliSearch logs?

I would like to know the size the 6898 entries take on disk, uncompressed, please? As little advice it is, I highly recommend you to index all your documents in one batch.

Thank you for helping us debug.

[shyim]

Log: https://drive.google.com/uc?id=1XXFA3YbBrtwMrhvsp9gRUqQ8uTO-pgs6

The data.ms folder is 20.9 GB 😱. I guess the deletion of index and recreating every hour does not cleanup anything 😅

After restart it looks normal again

[curquiza]

Hello @shyim and everyone following the issue!

The first RC of MeiliSearch v0.21.0 is out.
We did our best to fix indexation and crash issues. We succeeded to improve it, but not to totally fix them.

You can test this new release by downloading the binaries available in this release.
Or you can use it with docker:

docker run -p 7700:7700 getmeili/meilisearch:v0.21.0rc1 ./meilisearch

We will still improve this after the release of the v0.21.0. We would rather release a non-completely optimized version rather than delay it and, at the same time, delay the release of new features. Be sure we are doing our best to always improve these indexation issues.

As a reminder:

• despite the improvements, we always recommend pushing your documents to MeiliSearch by batch and not one by one. The maximum payload size of MeiliSearch is 100Mb and can be optionally increased. It means most of the dataset can be pushed with one call to the API.
• If you still have a memory crash, it means the RAM of your machine might not be adapted to your dataset size. We recommend increasing the RAM of your machine.

Thanks for your patience and your help with this! ❤️

[shyim]

Looks better. Thanks!