-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add blocked quantization mode for De/QuantizeLinear (#5812)
### Description Blocked quantization divides input tensors into smaller 1-D blocks that share the scale and zero-point. Scale and zero point should have the same rank as the input tensor. ### Motivation and Context Blocked quantization (sometimes referred to as group-quantization) is described in numerous papers. By allowing finer granularity of the quantization parameters, accuracy results improve, even under extreme compression factors. Blocked quantization is an inherent part of the Microscaling (MX)-compliant data formats. While MX-types are not yet adopted by the ONNX standard yet, adding support for blocked quantization is a first step in this direction. References: [OCP Microscaling Formats (MX) Specification v1.0](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf) [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://arxiv.org/pdf/2306.00978.pdf) [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323) [8-bit Optimizers via Block-wise Quantization](https://arxiv.org/abs/2110.02861) --------- Signed-off-by: Gal Hubara Agam <ghubaraagam@nvidia.com>
- Loading branch information
Showing
24 changed files
with
1,041 additions
and
140 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.