Repository navigation
Applying a Custom Quantization Method (like BFP) for QAIRT/NPU Deployment #4048
Unanswered
Changhyeon Lee (2changhyeon)
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi all,
I have a question about applying custom quantization techniques for deployment on Qualcomm NPUs using the QAIRT toolchain.
I would like to apply a custom quantization method to specific layers, which is not one of the default quantization methods provided by QAIRT (like standard min/max affine quantization).
Specifically, for a nn.Linear layer, I want to apply Block Floating Point (BFP) quantization to its inputs and weights, instead of the standard min/max quantization.
My current approach is to create a custom PyTorch module (let's call it BFPLayer) that simulates the BFP logic (finding shared exponents, quantizing mantissas, etc.) directly within its forward pass.
My questions are:
When using a tool like AIMET's QuantizationSimModel to prepare the model, is the correct approach to treat this BFPLayer as already quantized? Should I just call QuantizationMixin.ignore(BFPLayer) and let the tool skip over it?
My main concern is about NPU compatibility. This new quantization method (BFP) is simulated in PyTorch using a complex graph of basic operations (e.g., log2, pow, floor, bmm, sum, etc.), which gets exported to ONNX. Will this complex graph cause issues during compilation for an NPU, which only supports a limited set of optimized operations?
Will the QAIRT compiler fail to understand this complex BFP simulation graph? Or, is there a risk that it compiles, but runs very inefficiently because the simulation operations don't map to optimized hardware instructions?
What is the recommended workflow for applying a completely custom, non-affine quantization scheme (like BFP) to certain layers and successfully compiling the model for a Qualcomm NPU?
Thank you for your guidance.
All reactions