Threadpool: take 2

[fmz]

Added an API to support explicit management and fine-grain control of threadpools.
The API supports creating different threadpools for various parts of execution, e.g. batch, single-token, etc. Each threadpool can be created, paused, resumed, and released independently from any other threadpools. This mitigates the overhead of starting/stopping threads for each decode call and helps OSes keep track of scheduling history in order to make better scheduling decisions.

Each threadpool supports:

Setting number of threads (duh)
Setting a CPU mask for threads to be placed on
Support for strict/relaxed placement: pinning specific threads to specific cores, or letting the OS decide
Support for polling/interrupt-driven wait
Setting thread priority
Using threadpools explicitly is optional. If a llama_decode is called with a llama_context that doesn't have a threadpool attached, a disposable threadpool is created (same as the current behavior).
If users choose to explicitly use threadpools, they have to manage them manually. See examples in main.cpp and in speculative.cpp.

With all the bells and whistles enabled, we generally see .25-1 tok/s improvement across the board.

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[fmz]

Revert all of this

[fmz]

On W-2225 Xeon machine: CPU backend:

CPU	Model	Test	t/s master	t/s threadpool-attempt-2	Speedup
Intel(R) Xeon(R) W-2225 CPU @ 4.10GHz	llama 7B Q4_0	pp512	17.57	17.56	1.00
Intel(R) Xeon(R) W-2225 CPU @ 4.10GHz	llama 7B Q4_0	tg128	6.93	7.14	1.03

$ ./build/bin/llama-bench -t 8

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	8	pp512	18.38 ± 0.46
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	8	tg128	8.37 ± 0.01

[fmz]

./scripts/compare-commits.sh master threadpool-attempt-2 -t 1,2,4,6,8,10

CPU	Model	Threads	Test	t/s master	t/s threadpool-attempt-2	Speedup
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	1	pp512	3.93	3.94	1.00
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	1	tg128	2.43	2.44	1.00
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	2	pp512	7.13	7.06	0.99
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	2	tg128	4.37	4.36	1.00
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	4	pp512	11.96	11.99	1.00
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	4	tg128	6.79	6.77	1.00
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	6	pp512	14.96	14.98	1.00
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	6	tg128	7.51	7.53	1.00
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	8	pp512	13.06	13.09	1.00
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	8	tg128	6.88	6.83	0.99
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	10	pp512	14.08	14.06	1.00
Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz	llama 7B Q4_0	10	tg128	7.49	7.52	1.00

[fmz]

$ LLAMA_CUDA=1 ./scripts/compare-commits.sh master threadpool-attempt-2 -nkvo 0,1
(No discernible difference between OpenMP and ggml_threadpool with default settings)

GPU	Model	NKVO	Test	t/s master	t/s threadpool-attempt-2	Speedup
RTX 3060 Laptop GPU	llama 7B Q4_0	No	pp512	1644.73	1642.34	1.00
RTX 3060 Laptop GPU	llama 7B Q4_0	No	tg128	65.94	65.89	1.00
RTX 3060 Laptop GPU	llama 7B Q4_0	Yes	pp512	287.28	286.44	1.00
RTX 3060 Laptop GPU	llama 7B Q4_0	Yes	tg128	54.56	54.32	1.00