Small GCC 12 vs Clang 18 CPU test

[USBhost]

Observations: Clang does not like llama.cpp fp16/Q8_0 at least with my CPU (EPYC 7F72). Going with stock make with clang we have .08 t/s slower inference and 8ish t/s slower prompt processing. Ofast however fixed inference speed to be the same as GCC. However when using K quants it's faster by a respectable amount. See the K test section to see how it runs better. So normal Q8/fp16 is faster on GCC However when you use K quants it's faster.

Note: For GCC with or without LLAMA_FAST made almost no significant difference. Just like .1 faster pp (in the margin of error).

./llama-bench -m /mnt/36TB/AI/llama-3-8B-Instruct-abliterated/ggml-model-f16.gguf -t 24 -r 5 -pg 512,128

Clang 18 make DEFCC=clang-18 DEFCXX=clang++-18 -j:

model	size	params	backend	threads	test	t/s
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp512	50.69 ± 0.29
llama 8B F16	14.96 GiB	8.03 B	CPU	24	tg128	8.83 ± 0.01
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp512+tg128	25.95 ± 0.09

With: LLAMA_FAST=1 aka with Ofast vs O3

model	size	params	backend	threads	test	t/s
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp512	50.66 ± 0.17
llama 8B F16	14.96 GiB	8.03 B	CPU	24	tg128	8.88 ± 0.01
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp512+tg128	25.98 ± 0.06

GCC 12 make -j:

model	size	params	backend	threads	test	t/s
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp512	58.50 ± 0.37
llama 8B F16	14.96 GiB	8.03 B	CPU	24	tg128	8.88 ± 0.01
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp512+tg128	27.47 ± 0.06

Clang 18:
./llama-bench -m /mnt/36TB/AI/llama-3-8B-Instruct-abliterated/ggml-model-f16.gguf -t 24 -r 5 -p 64,128,265,512,768,1024 -n 0

model	size	params	backend	threads	test	t/s
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp64	67.08 ± 0.16
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp128	69.39 ± 0.08
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp265	67.34 ± 0.09
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp512	50.65 ± 0.60
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp768	54.68 ± 0.23
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp1024	49.93 ± 0.25

GCC 12:
./llama-bench -m /mnt/36TB/AI/llama-3-8B-Instruct-abliterated/ggml-model-f16.gguf -t 24 -r 5 -p 64,128,265,512,768,1024 -n 0

model	size	params	backend	threads	test	t/s
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp64	87.09 ± 0.37
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp128	89.85 ± 2.07
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp265	83.74 ± 1.00
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp512	58.76 ± 0.54
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp768	65.02 ± 0.75
llama 8B F16	14.96 GiB	8.03 B	CPU	24	pp1024	57.60 ± 0.32

################ K Quant TEST ################

GCC 12:

./llama-bench -m /mnt/36TB/AI/llama-3-70B-Instruct-abliterated/ggml-model-Q6_K.gguf -t 24 -r 3 -p 64 -n 64

model	size	params	backend	threads	test	t/s
llama 70B Q6_K	53.91 GiB	70.55 B	CPU	24	pp64	6.23 ± 0.00
llama 70B Q6_K	53.91 GiB	70.55 B	CPU	24	tg64	2.27 ± 0.00

./llama-bench -m /mnt/36TB/AI/llama-3-70B-Instruct-abliterated/ggml-model-Q8_0 -t 24 -r 3 -p 64 -n 64

model	size	params	backend	threads	test	t/s
llama 70B Q8_0	69.82 GiB	70.55 B	CPU	24	pp64	9.45 ± 0.01
llama 70B Q8_0	69.82 GiB	70.55 B	CPU	24	tg64	1.86 ± 0.00

./llama-bench -m /mnt/36TB/AI/llama-3-70B-Instruct-abliterated/ggml-model-fp16.gguf -t 24 -r 3 -p 64 -n 64

model	size	params	backend	threads	test	t/s
llama 70B F16	131.42 GiB	70.55 B	CPU	24	pp64	8.75 ± 0.00
llama 70B F16	131.42 GiB	70.55 B	CPU	24	tg64	0.96 ± 0.00

Clang 18

./llama-bench -m /mnt/36TB/AI/llama-3-70B-Instruct-abliterated/ggml-model-Q6_K.gguf -t 24 -r 3 -p 64 -n 64

model	size	params	backend	threads	test	t/s
llama 70B Q6_K	53.91 GiB	70.55 B	CPU	24	pp64	7.48 ± 0.00
llama 70B Q6_K	53.91 GiB	70.55 B	CPU	24	tg64	2.26 ± 0.01

./llama-bench -m /mnt/36TB/AI/llama-3-70B-Instruct-abliterated/ggml-model-Q8_0 -t 24 -r 3 -p 64 -n 64

model	size	params	backend	threads	test	t/s
llama 70B Q8_0	69.82 GiB	70.55 B	CPU	24	pp64	8.80 ± 0.03
llama 70B Q8_0	69.82 GiB	70.55 B	CPU	24	tg64	1.85 ± 0.00

./llama-bench -m /mnt/36TB/AI/llama-3-70B-Instruct-abliterated/ggml-model-fp16.gguf -t 24 -r 3 -p 64 -n 32

model	size	params	backend	threads	test	t/s
llama 70B F16	131.42 GiB	70.55 B	CPU	24	pp64	6.73 ± 0.00
llama 70B F16	131.42 GiB	70.55 B	CPU	24	tg32	0.96 ± 0.00