question: How can I quantize BERT to FP16 ?

[hexiaoyupku]

I have only P100 and V100 which dosen't support INT8. So what should I do to quantize BERT to FP16 ?
Thanks in advance!

[ofirzaf]

Our framework for the time being only simulates quantized inference, meaning, that the quantized GEMM operations are still done using FP32 arithmetic with integer values.
We don't have quantization aware training for FP16 implemented.
What you can do is train BERT regularly in FP32 and than convert the model to FP16 by calling half() method on the model.