Does this framework support "what you serve is what you train" for weight only quantization?

[chenho74]

Keeping activations as float while quantizing the weights to int8 or int4. I see that currently how it is done is by using float * float dot product. is there a int * float dot product available?

[lukaszlew]

Yes, in the config, you can select it individually.

[chenho74]

Thanks for the speedy reply. Would you mind pointing me to the config field that toggles this?

[lukaszlew]

https://github.com/google/aqt/blob/main/aqt/common/aqt_config.py#L362

As you see lhs and rhs have completly separate quantization configs.

[chenho74]

Yes, but from what i see in the underlying dot product code, if activation is not quantized, a float * float dot product is used? https://github.com/google/aqt/blob/main/aqt/jax/aqt_dot_general.py#L91 Is this a fake quantization or is this also the arithmetic in serving time?

[lukaszlew]

Sorry that I missed your comment.
For the dot product to be accelerated, both sides need to have the same type.
You are probably most concerned about weight loading.
We will implement that in AQTv2 within a month.