Skip to content

Commit

Permalink
Running small but cpu intensive tasks in a thread pool.
Browse files Browse the repository at this point in the history
This PR removes usage of spawn_blocking for cpu intensive tasks.

The problem with spawn_blocking is that these tasks get scheduled on a
ever growing thread pool.

For instance, when the server is under load, the GZIP decompression of
payloads could considerably increase the load factor of quickwit,
possibly making it unresponsive to healthcheck.

This PR isolates the thread pool used for the searcher, and instantiates
a second generic thread pool dedicated to those short cpu-intensive tasks.
  • Loading branch information
fulmicoton committed May 10, 2024
1 parent 8d69afc commit 006eb73
Show file tree
Hide file tree
Showing 21 changed files with 370 additions and 189 deletions.
20 changes: 10 additions & 10 deletions quickwit/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 006eb73

Please sign in to comment.