What exactly is the nf4 data type?

[AlpinDale]

There seems to be no proper explanation for the data type anywhere I've seen (maybe I'm not looking hard enough). What exactly is this data type, and how is FP4 normalized to get NF4?

[r3curs10n]

nf4 is the normalized float 4 bit data type. It quantizes floats into 4 bits. This video (at the specific timestamp) explains it in more detail: https://youtu.be/TPcXVJ1VSRI?t=563

[TimDettmers]

See also the QLoRA paper: https://arxiv.org/abs/2305.14314

I will see to add documentation somewhere for people that are interested to learn more.

[TimDettmers]

I now added documentation in the commit above. Thanks again for reporting the lack of information on this.

[RonanKMcGovern]

I see, so instead of evenly distributing the quantized levels, they are normally distributed. Interesting the optimum is a normal distribution. Is this because language model weights are normally distributed - kind of surprises me?

Interesting it's not exactly symmetric as well, those 16 datapoints