Will there be a difference if NF4 is 130 or 121 (sign bit, exponent, mantissa)?

[mrgalper]

If the quantile bin values remain the same regardless of the format, does it even matter which data type we use, since the binary representation only determines which bin the value falls into?

[TimDettmers]

Closing this issue. NF4 is not a floating-point type — it does not have a sign/exponent/mantissa decomposition. NF4 is a lookup table of 16 values derived from quantiles of the standard normal distribution. The 4-bit index simply selects from this lookup table. Whether you think of the 4 bits as "1-3-0" or "1-2-1" doesn't matter because the bits are just an index, not a float encoding. This is fundamentally different from FP4 (which is an actual floating-point format).

We are improving the documentation to make this distinction clearer.

[TimDettmers]

The distinction between NF4 (lookup table, no float encoding) and FP4 (actual floating-point format) is now documented in commit d7c179c — see the inline comments in get_4bit_type and the docstring on create_normal_map.