Would Per-Tensor Int8 KV Cache Break Small LLMs?

[Leo Wang (dw61)]

We’ve observed that the current KV cache uses per-tensor int8 quantization, but each KV tensor is large (992×64). Wouldn’t this lead to significant precision loss, especially in small models like a 0.5B LLM, where parameter sensitivity is high? Has Qualcomm encountered performance issues from this setup?

Also, for activations, are there alternative quantization methods beyond per-tensor?

[Kyunggeun Lee (quic-kyunggeu)]

Hi Leo Wang (@dw61), sorry for delayed resopnse. This is a very good question 😊

According to our observation, small LLMs weren't particularly more sensitive to int8 KV cache.
With int8 KV cache, small LLMs (<=1B) have shown mostly decent accuarcy.
We did observe some models can be sensitive to int8 KV cache, but it was due to the model characteristics, not model size.