how to convert int8 converted onnx model to tensorrt?

[lucasjinreal]

Hi, I followed the example and successfully converted the model to int8 on GPT2 model.
but the model generated with some Customized onnx op:

Shape,Gather,Range,Unsqueeze,Concat,Reshape,Add,LayerNormalization,DynamicQua │
│ ntizeLinear,Slice,Mul,MatMulInteger,Squeeze,Cast,Split,Sub,Transpose,MatMul,Pow,Div,Wh │
│ ere,Softmax,FastGelu,SkipLayerNormalization

Such as DynamicQuantizeLinear, FastGelu, how to converted it to tensorrt?

the int8 model was 400M compares with original 1.2G. much more small, if can inference via tensorrt, it can be massively accelerated.

[ykim362]

In this repository, we demonstrate CPU quantization only. For the GPU quantization, please visit https://github.com/Microsoft/onnxruntime. Thank you for your interest.