ikawrakow censorship

[cora4]

.

[ikawrakow]

Which model is this?

[ikawrakow]

For Gemma4-26B less than 2/3 of the tensors have a row size of 704. You increase the model size by about 2 GiB compared to IQ3_K.

But yes, it is what it is. In the early days when I was still contributing to llama.cpp I was maintaining a version of K-quants with a block size of 64. But then we decided to drop it as Falcon was the only model that needed it.

[ikawrakow]

Well, yes. The way bits get packed and unpacked absolutely depends on the block size, so going to a block size of 64 is basically a new quantization type.

[ikawrakow]

IQ4_KS, IQ4_KS_R4 and IQ5_KS_R4 quants can be easily made to work for row sizes that are only a multiple of 32. It is just a matter of handling the tail when the row size is not divisible by 256. The divisible by 256 requirement can also be removed from the Trellis quants by adding appropriate tail handling.

Do you want to ask your favorite LLM to do it?

[ikawrakow]

Haha, really? Everyday I come across several articles/blogs/whatever telling me how great the code is that LLMs write and how one is 10X more productive, and how soon we all will be replaced by AI.