v0.29: Updates to auto-batching

[maryamsulemani97]

With v0.29, auto-batching is enabled by default.

• Remove the Experimental section from the docs
• Move Auto-batching somewhere else. Maybe Advanced topics. Does it even need its own page now?
• Update the page to remove the current options: --debounce-duration-sec, --max-batch-size, and --max-documents-per-batch
• Remove batchUid from the task object as it is now an internal field
• Add new instance option: --disable-auto-batching and environment variable: MEILI_DISABLE_AUTO_BATCHING
• Mention auto-batching in quick start

References:

• Spec: Auto-batching - Enable feature by default and remove unwanted options specifications#162
• Issues on core:
  • Remove the batchUid field from the tasks route meilisearch#2676
  • Auto-batching: improve error handler + make it available by default meilisearch#2582
• SME: @Kerollmops

[curquiza]

Pinging @Kerollmops who can be a better SME than me on it 😇

[curquiza]

Also, we can remove the recommendation about indexing by batch here: https://docs.meilisearch.com/learn/advanced/indexing.html#improving-indexing-performance

[guimachiavelli]

Two things that might be relevant when working on this issue:

1. Change the behavior of auto-batching error catching meilisearch#2478 might be worth a quick read
2. Not sure if we should mention it (probably more of a changelog thing), but FYI launching meilisearch v0.29 with the current options will throw an error

[curquiza]

Not sure if we should mention it (probably more of a changelog thing), but FYI launching meilisearch v0.29 with the current options will throw an error

Since this feature was in beta this is something not mandatory to write. My opinion is we handle this on the changelog side and on the associated discussion side, but not necessarily on the docs side. However, I don't have a strong opinion, and both are correct to me. So, it's up to you of course 😇

[guimachiavelli]

Yeap, agreed. I'll leave the final decision to @maryamsulemani97, just wanted her to be aware of it since we did briefly discuss it during the opensource meeting.

[maryamsulemani97]

@Kerollmops
Can you please describe a situation where disabling auto-batching might be useful? We would like to add a sentence or two on what users can gain from disabling this feature.

[Kerollmops]

@maryamsulemani97, one of the reasons I currently see where it would be helpful to disable the auto-batching system would be to:

1. Avoid a potential bug in the auto-batching system itself.
2. Reduce the visibility latency. If the engine indexes documents in big chunks it will take more time to be able to see the documents of the first task in a search query. If you disable the auto-batching, the whole queue will take much more time to be processed, but each individual task will be available earlier (until a certain amount of processed tasks).