can quantized onnx t5 run on GPU

[gyin94]

Just wonder whether this quantized onnx t5 can run on GPU.

[Ki6an]

unfortunately no. quantized onnx models can only be run on CPU & onnxruntime-gpu does not support quantization.
if you want more details on this question you should create an issue in onnxruntime.

[mblank5]

@Ki6an HI, do we support onnxruntime-gpu mode with V100 or T4 if dont use the quantization?

i see with onnxruntime only we can have x2 speedup ?

[Ki6an]

@mblank5 no, this library uses onnxruntime, and to support GPUs you need to have onnxruntime-GPU installed.

BUT you can uninstall onnxruntime after the fastt5 library is installed, and install onnxruntime-gpu and try running the model but not sure you'll get speed up. for more info refer to this issue.

with onnxruntime, you'll get speed up if you are using modern CPUs and more CPU cores. refer benchmark section of README.