INT4 and other low-precision conversion support status

[AIWintermuteAI]

What is the current status of model conversion (PTQ specifically) with INT4 precision?

The question was raised before here #60125.
Also it looks like INT4 support is being added to various parts of tensorflow, as evidenced by #63870 and

tensorflow/tensorflow/compiler/mlir/quantization/tensorflow/quantization_options.proto

Line 76 in 6738c28

TENSORTYPE_INT_4 = 1;

.

However at the moment there seems to be no way to quantize model to INT4 (specifically the weights):

• the tflite converter MLIR based conversion (the "experimental" conversion) seems abandoned. It is not possible to specify the type of weights quantization
• what seems to be a new API for quantization (is it the one from blog article https://blog.tensorflow.org/2023/05/google-io-2023-whats-new-in-tensorflow-and-keras.html ?) lives here https://github.com/tensorflow/tensorflow/tree/master/tensorflow/compiler/mlir/quantization/tensorflow (???). It is possible to use it with INT8, but not INT4 - but the resulting SavedModel cannot be converted to .tflite (SegFault).

Can anyone who actively works on this in TF team shine the light on what is the current direction and where one needs to dig to add INT4 PTQ quantization?

[LakshmiKalaKadali]

Hi @AIWintermuteAI,

Please share your .tflite file to reproduce the issue.

Thank You

[github-actions]

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

[pkgoogle]

Hi @abattery, can you please take a look or comment on this? Thanks.