Is there no tutorial for dynamic quantization of BERT model in torch.ao?

[Esttelle]

I saw that some quant related tutorials in the PyTorch tutorials repo have been deleted, and the PR stated that these tutorials will be moved to torchao. However, I can't find the BERT dynamic quantization tutorial in the torchao repository. Where can I find it?

[jerryzh168]

Hi @Esttelle we don't have a equivalent tutorial in torchao, but we do support the same functionality with

from torchao.quantization import quantize_, Int8DynamicActivationInt8WeightConfig
quantize_(model, Int8DynamicActivationInt8WeightConfig())
model = torch.compile(model)

This should work for both CUDA and CPU, although we haven't tested the performance for CPU, if there is need for CPU optimization, maybe @Xia-Weiwen can help take a look I think