Batch document additions and deletions together

[loiclec]

When someone sends many document addition tasks interspersed with document deletion tasks, meilisearch is currently forced to process these tasks serially. For example, if the following tasks are sent in this order:

1. document addition
2. document addition
3. document addition
4. document deletion
5. document deletion
6. document addition

Then they will be batched as follows:

a. (1–3) document additions
b. (4–5) document deletions
c. (6) document addition

This can cause significant indexing performance problems, as we rely on incremental indexing speed to keep up with the updates. If the updates are sent quicker than meilisearch can process then, then the task queue will keep growing bigger and bigger.

Ideally, we want to batch all of these tasks into:

a. (1–6) document additions and deletions

---

Proposed solution

We could add a new function in milli which can accumulate document additions and deletions, respecting the order of the operations. The output of this function should be two things:

1. A set (roaring bitmap) of documents to delete
2. A Transform or TransformOutput containing the documents to add

Then we can use the existing indexing functions to process (1) and (2) serially.

---

impacted team

Can impact @meilisearch/docs-team in some part of the docs talking about auto batching

[gmourier]

Thank you, @loiclec!

In the future, (not saying it will be needed), could we leverage that enhancement to provide an endpoint permitting sending hybrid documents ops akka containing additions (updates/replace) and deletions in the same HTTP request?

I see it as a friction reducer in the case you receive a batch of ops (from a message queue or a file system) being mixed and you are forced to separate them by type before sending them to Meilisearch.

[loiclec]

@gmourier Sure, it's technically possible :) The main challenge would be in making a good API for it.

[gmourier]

Thank you @loiclec

It could be related to meilisearch/product#554

[curquiza]

We will try to be investigated and maybe done during v1, no guarantee however

[irevoire]

After an unexpected and hard-to-find/debug bug on the implementation, we decided to cancel this feature: #3667

We'll work on it again later.

[gillian-meilisearch]

Hello everyone 👋

We have just released the first RC (release candidate) of Meilisearch containing this fix!
You can test it by using

• the release assets
• the Meilisearch Docker image

docker run -it --rm -p 7700:7700 -v $(pwd)/meili_data:/meili_data getmeili/meilisearch:v1.3.0-rc.0

If you encounter any bugs, please report them here.
Thanks in advance for your help and your involvement in Meilisearch ❤️

🎉 The official and stable release containing this change will be available on July 31st, 2023

⚠️ RC (release candidates) are not recommended for production