Question about the Q8_0 quants

[RandomGitUser321]

@city96 I noticed that the data in the flux dev and schnell Q8_0 ggufs are in f16/q8_0, but shouldn't it be f32/q8_0?

Flux in Q8_0:

Flux in Q6_K:

Here's an example of a Llama3.1 gguf in Q8_0:

I also checked your t5 Q8_0 gguf and it's using f32 and Q8_0. Is there some kind of reason for the dev/schnell quants being in f16/Q8 instead of f32/Q8?

[city96]

Good observation. The _0 and _1 legacy quants were done day1 before we had C++ code for quantizing and before the logic was finished. They should indeed be F32, though most likely there shouldn't be much difference even with F16. I'll get around to reuploading them eventually.

[RandomGitUser321]

Yeah I kind of figured that might have been the case, but you're right, there wouldn't likely be that much difference between those being F32 instead of F16. I was just curious if it had to do with loading them or compatibility or something along those lines. Thanks again for all the hard work!

[AfterHAL]

@city96 I noticed that the data in the flux dev and schnell Q8_0 ggufs are in f16/q8_0, but shouldn't it be f32/q8_0?

Hi @city96. May I ask what's the tool you use to get those informations from the blocks ?
Thank you.

[city96]

@AfterHAL It's the preview you get on huggingface when you click this button

[AfterHAL]

@AfterHAL It's the preview you get on huggingface when you click this button
Thank a lot @city96 !

[blepping]

GGUF (gguf-py directory in llama.cpp) comes with some tools also, there's a gguf_dump.py script which will show all the tensors and metadata. so you can also get access to that information from the commandline if you want.