Maximum index size with good performance?

[rectifyer]

We have a lot of data that would be spread across different indexes such as Movies, Shows, and Episodes. One of these currently has over 45 million documents in our SQL database and all of them would need to be searchable (title + overview at minimum) but also contain a lot of the other metadata for display purposes.

Does anyone have benchmarks or experience with indexes this large? I know there are many other variables that come in to play, but I'm just trying to find out a high level if Meilesearch can work at this type of scale? In addition to performance, I'm also not sure if this amount of data is more disk or more memory intensive?

Thanks for any insight or articles about this.

[ManyTheFish]

Hello @rectifyer,
First of all, we have a dataset of songs containing 110M documents. This data fits in Meilisearch, and the search response time is marginally over 100ms on a Meilisearch-cloud PRO plan. However, each document is small and has around four searchable fields.

We have a lot of data that would be spread across different indexes such as Movies, Shows, and Episodes

It is a good thing to do. Moreover, if you want to search over all your data globally, you can do it in 1 API call using the multi-index search

One of these currently has over 45 million documents in our SQL database and all of them would need to be searchable (title + overview at minimum) but also contain a lot of the other metadata for display purposes

You can easily separate your searchable fields and the field you want to display. Don't hesitate to restrict the searchable field to the most relevant field from the search-user POV and prefer using the filterable fields for the attribute containing numbers. This will both enhance the relevancy of the search by removing the noisy fields and reduce the processing time.

Does anyone have benchmarks or experience with indexes this large? know there are many other variables that come in to play, but I'm just trying to find out a high level if Meilesearch can work at this type of scale?

This highly depends on the data shape and the used settings, it's difficult to forecast the performance. But it should work, and if it doesn't, come back to me, and we will fix it.

In addition to performance, I'm also not sure if this amount of data is more disk or more memory intensive?

Meilisearch is more disk-intensive than memory-intensive, but memory can be a limitation when trying to index a lot of data at once. On the search side, the search throughput is mainly related to the number of available CPUs.

On top of that, I suggest trying Meilisearch on your side, using the last release candidates (beta version), including several performance improvements. V1.6 will be stabilized next week.

I hope my answer was satisfying, don't hesitate to ask more question if you need them,
see you!

[rectifyer]

Thank you for the detailed response, I appreciate that! That's great to hear such a large dataset is so responsive. I plan to store a lot of fields per document, but only need 2 or 3 fields searchable (title, overview, tagline) so it sounds like that can work well. I am working with the meilisearch-rails gem and already have the indexing setup and working well, with the only the specific searchable fields.

I have been working locally with 1.5, but 1.6 does have a lot of great improvements after reading the release notes. I'll start using that locally while I'm in development. The plan is to self on AWS. We currently use Solr on a m5a.24xlarge (384GB ram, 96 vCPUs) but I plan on running Meilisearch in parallel so we can compare the live dataset.

I'm sure I'll have some follow ups and dev continues, thanks again!

[rectifyer]

I've moved into a large scale test, using our production data and so far it's going very well!

One question I have is around indexing speed. I'm running a m5a.4xlarge (64GB RAM, 16 CPU cores) and it has barely any load on the server, but the task queue sometimes grows to over 10,000 items. I'm using https://github.com/meilisearch/meilisearch-rails so it's only sending single documents as they get updated.

Are there are Meilisearch config settings to increase the task processing? I've tried explicitly setting the CPU and memory settings, but it didn't seem to have any effect.

[curquiza]

Hello @rectifyer

I answer because Many is on Holiday 😊

You can see a lot of items in the tasks queue because Meilisearch takes time to process them. Here are some tips to help Meilisearch process them faster.

Also, Meilisearch has an internal auto-batcher. It means Meilisearch can process multiple items in the task queue at the same time. But to happen, some conditions have to be met, like being two consecutive tasks related at the same index. If you update two different tables in your SQL DB, maybe it's related to two different indexes in Meilisearch, so cannot be auto-batched

It will depend on your needs and usage. If you need your tasks in the task queue to be processed as soon as possible, maybe you should configure the way meilisearch-rails manage indexing by default to only update Meilisearch when necessary
If you don't need to, maybe the tips I shared to improve indexing performance, in general, would be enough and will reduce the number of items in the queue

[rectifyer]

Thank you for the reply! I'll do some reading and test out some of these ideas. I think the main issue might be your first point, the tasks are across 7 different indexes and most likely very random in order.