DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
PACT: Parameterized Clipping Activation for Quantized Neural Networks
Pact is used to replace relu activation function, not in convolution
If using Pact in QuantConv2d, the precision will be very slow
The loaded network is trained with the proposed quantization scheme in which ReLU is replaced with the proposed parameterized clipping ActFn for each of its seven convolution layers.
2023-01-08, upload dorefa、pact
I'm not the author, I just complish an unofficial implementation of dorefa and pact.
pytorch==1.11.0+cu113
You should train 32-bit float model firstly, then you can finetune a low bit-width quantization QAT model by loading the trained 32-bit float model
Dataset used for training is CIFAR10 and model used is Resnet18 revised
| version | learning rate | batchsize | Accuracy | models |
|---|---|---|---|---|
| Float 32bit | <=66 0.1 <=86 0.01 <=99 0.001 <=112 0.0001 |
128 | 92.6 | https://www.aliyundrive.com/s/6B2AZ45fFjx |
| dorefa | <=31 0.1 <=51 0.01 <=71 0.001 |
128*7+30 | 95 | https://www.alipan.com/s/WhZUqDUh4UB |
| pact | <=31 0.1 <=51 0.01 <=71 0.001 |
128*7+30 | 95 | https://www.alipan.com/s/F7ocFVSZwMb |
https://github.com/ZouJiu1/LSQplus
https://github.com/666DZY666/micronet
https://github.com/hustzxd/LSQuantization
https://github.com/zhutmost/lsq-net
https://github.com/Zhen-Dong/HAWQ
https://github.com/KwangHoonAn/PACT
https://github.com/Jermmy/pytorch-quantization-demo