Result with the branch `fastest-inference-4bit`

[alanxmay]

Amazing 4bit inference speed!

I happened to see a new branch fastest-inference-4bit, so I did a test:

LLaMA-13B	branch	Bits	group-size	memory(MiB)	PPL(c4)	Median(s/token)
FP16	fastest-inference-4bit	16	-	26634	6.96	0.0383
GPTQ	triton	4	128	8590	6.97	0.0551
GPTQ	fastest-inference-4bit	4	128	8699	19069	0.0344

But why the value of PPL is so large?

---

update result:

1. The act-order parameter is consistent in build compressed model and the benchmark
2. with groupsize = -1 and act-order = false:

LLaMA-13B	branch	Bits	group-size	memory(MiB)	PPL(c4)	Median(s/token)	act-order	speed up
FP16	fastest-inference-4bit	16	-	26634	6.96	0.0383	-	1x
GPTQ	triton	4	128	8590	6.97	0.0551	-	0.69x
GPTQ	fastest-inference-4bit	4	128	8699	6.97	0.0429	true	0.89x
GPTQ	fastest-inference-4bit	4	128	8699	7.03	0.0287	false	1.33x
GPTQ	fastest-inference-4bit	4	-1	8448	7.12	0.0284	false	1.44x

[qwopqwop200]

Unlike other branches, the act-order is important. Make sure you use act-order when creating the model.
Fastest but does not support act-order.
python llama.py ${MODEL_DIR} c4 --wbits 4 --groupsize 128 --load llama7b-4bit-128g.pt --benchmark 2048 --check
Supports act-order, but slightly slower.
python llama.py ${MODEL_DIR} c4 --wbits 4 --groupsize 128 --act-order --load llama7b-4bit-128g.pt --benchmark 2048 --check

[alanxmay]

Thanks, I'll try it

[alanxmay]

Updated results and found that act-order does have an impact on performance

[Qubitium]

@alanxmay Can you also do a 4bit perf test with groupsize -1 + act-order=false? There may be more speedup with no group-size. It will be good to know so others can see if they want speed vs quality vs memory usage.

[alanxmay]

@diegomontoya Ok, I will do a test later

[alanxmay]

@diegomontoya Done and saw a slight performance improvement

[Ph0rk0z]

So act order without groupsize won't work like before?

[joshlevy89]

@alanxmay Maybe a noob question but qhy is it that GPTQ is slower (.89x) than FP16? With lower memory consumption, I would have figured it'd be much faster.

[alanxmay]

@joshlevy89 I guess inference speed is more closely related to computational optimization. The GPTQ algorithm is only used to improve the accuracy of the quant model.

[Dhaladom]

@joshlevy89 I did not yet find the time to look at the GPTQ algorithm in depth, but from what I understand it compresses the weights (rather obviously). At inference time, some additional computation will have to be performed to retrieve the (near) original information. Hence, the slowdown.

[joshlevy89]

@Dhaladom thanks! it makes sense to me that there'd be some additional overhead...on the other hand, I would have expected that the benefits of working with 4bit vs 16bit would far outweigh that. Guess not though!