Add logs when upload task fails (#5699)

Updating tantivy in order to include the hotfix of the bug which was …

…causing a merge panic

... on fastfiedl columns containing a very large number of values.

Attempt to limit the impact of bug #5687.

With this commit, a failing merge will not restart the indexing pipeline.
This should prevent the split of death loop.

This also include a tantivy fix, that adds a panic handler on the merge thread pool

Fix `root_search_requests_total` metric (#5654)

 Limit and monitor warmup memory usage (#5568)

* Measure and log the amount of memory taken by a split search, and log this.

* Limit search memory usage associated with warmup.

Due to tantivy limitations, searching a split requires downloading all
of the required data, and keep them in memory. We call this phase
warmup.

Before this PR, the only thing that curbed memory usage was the search
permits: only N split search may happen concurrently.
Unfortunately, the amount of data required here varies vastly.

We need a mechanism to measure and avoid running more split search
when memory is tight.

Just using a semaphore is however not an option. We do not know
beforehands how much memory will be required by a split search, so it could easily
lead to a dead lock.

Instead, this commit builds upon the search permit provider.

The search permit provider is in charge of managing a configurable memory budget for this warmup memory.

We introduce here a configurable "warmup_single_split_initial_allocation".
A new leaf split search cannot be started if this memory is not
available. This initial allocation is meant to be greater than what will
be actually needed most of the time.

The split search then holds this allocation until the end of warmup.
After warmup, we can get the actual memory usage by interrogating the
warmup cache. We can then update the amount of memory held.
(most of the time, this should mean releasing some memory)

In addition, in this PR, at this point, we also release the warmup search permit:

We still have to perform the actual task of searching, but the thread
pool will take care of limiting the number of concurrent task.

Closes #5355

* Bring some clarifications and remove single permit getter

* Make search permit provider into an actor.

Also attach the permit to the actual memory cache to ensure memory is freed at the right moment.

* Revert weird cargo lock update

* Improve separation of concern by using wrapping instead of nesting

Adding an extra generic field into the cache to optionally allow permit tracking is weird.
Instead, we make the directory generic on the type of cache and use a wrapped cache
when tracking is necessary.

* Fix clippy

* Fix undefined incremental resource stat

* Add tests to permit provider

* Improve and test stats merging utils

* Fix minor typos

* Add test for permit resizing

* Increase default warmup memory

* Increase default warmup memory

* Add warmup cache metric

* Limit permit memory size with split size

* Also use num_docs to estimate init cache size

* Restore sort on HotCache file list

* Minor closure renaming

* Add minimum allocation size

* Increase default warmup memory to limit its effect

---------

Co-authored-by: Paul Masurel <paul@quickwit.io>

Return 429 on ES API when no shards available (#5566)

add histogram to more core s3 calls

and make their histograms appear only when the endpoint has been called

add histogram to more core s3 calls

and make their histograms appear only when the endpoint has been called

Log PostgreSQL metastore error (#5530)