Quantization via MLIR

[josevalim]

No description provided.

[jonatanklosko]

There are quantized tensor types in StableHLO spec, but looks like there are still discussions around it, for example openxla/stablehlo#1491. Also, that's only the specification, based on openxla/xla#9291 I infer it's not implemented in XLA yet. So on this level the best bet is to wait.

There is also google/aqt, which is actually the only quantization reference/project I found for Jax/Flax. It lifts quantization one level up, by introducing a quantized tensor/config and reimplementing some operations with quantization in mind.

Finally, it's worth noting that quantization is more broad, there are many approaches and algorithms, and both the use case and hardware plays a role. hf/transformers has a matrix of different techniques.