Allow Int4WeightOnlyQuantizer to set different dtype for scales_and_zeros #479
Conversation
scales_and_zeros As titled. Currently `Int4WeightOnlyQuantizer` is hardcoded to return `scales_and_zeros` with dtype `torch.bfloat16`. Adding `dtype` argument into the flow so that it can be different dtype.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/479
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit f3c320a with merge base a35a1cd ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
| ) -> None: | ||
| super().__init__() | ||
| self.padding = not _check_linear_int4_k(in_features, groupsize, inner_k_tiles) | ||
| if self.padding: | ||
| from model import find_multiple |
There was a problem hiding this comment.
I don't think there's a module called model
There was a problem hiding this comment.
Thanks I think this is a relic of when gptq was more deeply coupled with gpt-fast
|
This seems fine to merge although I do worry that most of our gptq tests are disabled right now in |
I'm migrating torchchat to use these APIs, to be prepared for shared kernels across ET and PyTorch eager/compile. |
As titled. Currently
Int4WeightOnlyQuantizeris hardcoded to returnscales_and_zeroswith dtypetorch.bfloat16. Addingdtypeargument into the flow so that it can be different dtype.