Quantisation step for canned t5-small model persistently fails (See details)

[PrithivirajDamodaran]

Executed the below code, it fails with the following error. Please advice.

Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from /content/models/t5-small-decoder-quantized.onnx failed:This is an invalid model. Error: Duplicate definition of name (pkv_11).

from fastT5 import (OnnxT5, get_onnx_runtime_sessions,
generate_onnx_representation, quantize)
from transformers import AutoTokenizer

model_or_model_path = 't5-small'

# Step 1. convert huggingfaces t5 model to onnx
onnx_model_paths = generate_onnx_representation(model_or_model_path)

# Step 2. (recommended) quantize the converted model for fast inference and to reduce model size.
quant_model_paths = quantize(onnx_model_paths)

# step 3. setup onnx runtime
model_sessions = get_onnx_runtime_sessions(quant_model_paths)

# step 4. get the onnx model
model = OnnxT5(model_or_model_path, model_sessions)

[Ki6an]

the latest version of onnxruntime is what causes the issue, you can fix it by downgrading the onnxruntime to 1.7.0 or below.

[PrithivirajDamodaran]

Works like a charm, Thanks Kiran!