Limit the number of search requests running concurrently

[irevoire]

Related product team resources: PRD (internal only)
Related product discussion: https://github.com/orgs/meilisearch/discussions/729
Related spec: WIP

Motivation

The search requests in meilisearch are intensive, which means it’s useless to run multiple searches at the same time.
The most efficient way to treat search requests is to run one search request per core.

But we don't want to refuse search requests straight away when all cores are being used; thus, we're going to keep track of the search requests to process in a queue of fixed-size and then randomly pick the next search request to be processed.

If the queue is full, then Meilisearch will return an error (TBD) with the error code 503 service unavailable and a retry-after header.

Since the size of the queue may vary between use cases, we'll also need to introduce a new global setting for the whole instance, specifying the size of the queue.

See full usage here: https://meilisearch.notion.site/API-usage-of-the-search-limiter-c1edc092f3c746d6899c5ba7bfa9a6a8?pvs=74

Usage

TODO

• merge changes into main
• Update the spec

Reminders when modifying the Setting API

• Ensure the new setting route is at least tested by the test_setting_routes macro
• Ensure Analytics are fully implemented
  • /settings/my-new-setting configurated in the make_setting_routes macro
  • global /settings route configurated in the update_all function
• Ensure the dump serializing is consistent with the /settings route serializing, e.g., enums case can be different (camelCase in route and PascalCase in the dump)

Special cases when adding a setting for an experimental feature

• ⚠️ API stability: The setting does not appear on the main settings route when the feature has never been enabled (e.g. mark it Unset when returned from the index in this situation. See an example)
• The setting cannot be set when the feature is disabled, either by the main settings route or the subroute (see validate_settings function)
• If possible, the setting is reset when the feature is disabled (hard if it requires reindexing)

Impacted teams

@meilisearch/docs-team