Align scale dtype with model precision in GPTQ #2403
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2403
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 7f74703 with merge base d506cc7 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
xiaowangintel
changed the title
|
@liangan1 @jerryzh168 please help to review this pr. |
xiaowangintel
force-pushed
the
xw/int4-gptq-scale-type-align
branch
from
a6f03f8 to
7f74703
Compare
Xia-Weiwen
added
the
topic: not user facing
| @@ -295,7 +295,7 @@ def __torch_function__( | |||
| SQNR(DQ, DQ_from_qtensor), | |||
| ) | |||
|
|
|||
| qparams2 = cls.get_qparams_func(W) | |||
| qparams2 = cls.get_qparams_func(W, W.dtype) | |||
There was a problem hiding this comment.
thoughts on changing https://github.com/pytorch/ao/blob/994a4ba6c869854fcaa6ca7e118fcbd75e6c28cc/torchao/quantization/utils.py#L335C5-L335C10 to use w.dtype by default?
There was a problem hiding this comment.
No. For gptq algorithm, the original implement for weight scales calculation does not specify scale dtype, and use default dtype=bfloat16 of get_groupwise_affine_qparams function. And it's not suitable for other model-precision. We want to align the scale data type with linear weight primitive type. Therefore, the modification passes a dtype parameter to get_groupwise_affine_qparams.
There was a problem hiding this comment.
you mean it's required for gptq algorithm to use hardcoded bfloat16 by default?
And it's not suitable for other model-precision.
what does this mean?
Summary:
For general usage, align the data type of scale with model precision Instead of the default use of bfloat16.