Support for low bit-width Quantization in TensorRT

[hachons]

I'm interested in knowing if TensorRT currently supports quantization for 2-bit and 4-bit models. Could you confirm the same for mixed precision as well? Additionally, I'd appreciate insights on accuracy metrics and any potential challenges associated with achieving accurate inference at such low bit-widths.

[zerollzeng]

We don't support INT2 AND INT4, but FP8 will be supported in the 9.x release. cc @ttyio

[ttyio]

closing since no activity for more than 3 weeks, thanks all!