[BUG] Int4 Quantized model inference is slower than Float32 model

[r4ghu]

Describe the bug
I am trying to run some performance benchmarks of an Int4 quantized model against the original float32 model. While trying to run inference comparisons, I observed that the Int4 model is running slower than the Float32 or BFloat16 models.

I did observe a significant decrease in RAM usage and a slight reduction in GPU/CPU power usage for Int4 models. Still, I am trying to understand why the overall inference is slower. Please clarify if this is an expected behavior.

To Reproduce
Any standard LLM. I simplified my experiment to run only on the encoder.

Include code snippet

# ...performed model warmup prior to this
for prompt in data:
    prompt = tokenizer.encode(prompt)
    mx.eval(prompt)
    tic = perf_counter_ns()
    output = model.encode(prompt)
    output_tokens = output.tolist() # -> forces mx.eval()
    toc = perf_counter_ns()
    elapsed = toc - tic

Expected behavior
I was expecting the average and 90th percentile inference time of the Int4 model to be faster than the Float32 model. But the behavior is the opposite.

Desktop (please complete the following information):

• OS Version: MacOS 14.3.1
• Version 0.7.0

Additional context
N/A

[awni]

It really depends on the model and the size of the prompt if the quantized operations will be faster. If the prompt is not that short it will make sense that fp16 is faster.. as it's not longer a bandwidth bound computation.

For token generation on the other hand (aka the decoder in your case), the quantized model should typically be a lot faster as it's much more bandwidth bound.