server: run sampling in a threadpool

[ngxson]

Overview

Ref discussion: #24843

• Add server_threadpool that spawn n-1 threads (plus one main thread)
• Add --threads-sampling argument to select the number of threads

Benchmark: RTX 5060 Ti + 12th Gen Intel(R) Core(TM) i7-12700KF

10 concurrent requests --> around ~5% speed improvement

Baseline:

0.24.072.468 I slot print_timing: id 25 | task 4 | n_decoded =    481, tg =  25.27 t/s, tg_3s =  25.04 t/s
0.24.072.839 I slot print_timing: id 26 | task 5 | n_decoded =    481, tg =  25.27 t/s, tg_3s =  25.04 t/s
0.24.073.215 I slot print_timing: id 27 | task 3 | n_decoded =    481, tg =  25.28 t/s, tg_3s =  25.04 t/s
0.24.073.574 I slot print_timing: id 28 | task 2 | n_decoded =    481, tg =  25.28 t/s, tg_3s =  25.04 t/s

PR:

0.24.029.817 I slot print_timing: id 25 | task 5 | n_decoded =    502, tg =  26.75 t/s, tg_3s =  26.52 t/s
0.24.029.818 I slot print_timing: id 26 | task 4 | n_decoded =    502, tg =  26.75 t/s, tg_3s =  26.52 t/s
0.24.029.820 I slot print_timing: id 27 | task 3 | n_decoded =    502, tg =  26.75 t/s, tg_3s =  26.52 t/s
0.24.029.821 I slot print_timing: id 28 | task 1 | n_decoded =    502, tg =  26.75 t/s, tg_3s =  26.52 t/s

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: no

[ngxson]

point for discussion: should sampling_task be a member of server_slot ?

[ggerganov]

Can put it in common_params_sampling

[ggerganov]

This is problematic because common_sampler_sample calls llama_context API that is not thread-safe.

Probably need a new overload for sampling from multiple samplers: common_sampler_sample(std::vector<common_sampler *> smpls, ...). And multi-thread just the llama_sampler_apply parts.

[ngxson]

hmm ok I think that will requires having a common_threadpool that can be shared across multiple sampling calls, so that we can also reuse it for MTP sampling if needed