Dynamic quantization of multilingual miniLM - output does not match float32 version. Onnxruntime 1.9.0

[jobergum]

Describe the bug
First thanks a lot for all your hard work on onnxruntime, I love it! To the issue at hand:

The quantized version of the original onnx model output does look anything like the float onnx model and also downstream accuracy suffers greatly. I'm thinking that something is off with regards to this particular model and quantization.

The model is a multilingual MiniLM model fine trained for sequence classification with a classification head. Model card is on huggingface hub.

The model when exported to onnx with float32 works as expected with very similar score and also downstream accuracy as when using torch, however when using quantization the model output score does not match the float versions and similar the downstream task accuracy drops significantly (next to random behaviour).

Urgency
None

System information

• MacOS X
• ONNX Runtime version: 1.9.0
• Python version: 3.8.5

To Reproduce
The model could be downloaded from the internet, see instructions in the following notebook which also demonstrates
the output score difference. See notebook

Expected behavior
I would expect that the output is roughly the same score as the float version. I've successfully quantized subword tokenized MiniLM with classification heads with little or no accuracy drop as compared to the float32 version, but having some trouble with this model so I'm reaching out for help 💯

[yufenglee]

Hi @jobergum, made some tuning like below and can get a relatively close score for the example. Will send a PR to provide an option to disable the quantization of MatMul without constant weight.

quantize_dynamic("msmarco_multilingual.onnx",
                 "msmarco_multilingual_int8_is.onnx",
                 weight_type=QuantType.QUInt8,
                 extra_options={'WeightSymmetric':False},
                 op_types_to_quantize=['MatMul'],
                 nodes_to_exclude=['MatMul_101', 'MatMul_195', 'MatMul_289', 'MatMul_383', 'MatMul_477', 'MatMul_571',
                                   'MatMul_96', 'MatMul_190', 'MatMul_284', 'MatMul_378', 'MatMul_472', 'MatMul_566']) 

[jobergum]

Thank you for your feedback @yufenglee! Really appreciate you taking time to look at this,

provide an option to disable the quantization of MatMul without constant weight.
Could you clarify this?

With the above exclusions the model size is only reduced from 428005084 to 395738272 so it's not a lot of compression compared to full quantization which reduces the model to 107150516.

[jobergum]

@yufenglee Do you believe the mentioned #9640 address the accuracy problem with this model?

[yufenglee]

The PR exposes MatMulConstBOnly, so you can leverage it and you don't need to specify the nodes to exclude:

quantize_dynamic("msmarco_multilingual.onnx",
                 "msmarco_multilingual_int8_is.onnx",
                 weight_type=QuantType.QUInt8,
                 extra_options={'MatMulConstBOnly':True}) 

onnxruntime/onnxruntime/python/tools/quantization/quantize.py

Line 296 in 3f5c1e1

MatMulConstBOnly = True/False: Default is False. If enabled, only MatMul with const B will be quantized.

Please try it and see if it help. In addition, you can try enabling per_channel.

[stale]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[snnn]

This issue has been automatically closed as 'not planned' because it has been marked as 'stale' for more than 30 days without activity. If you believe this is still an issue, please feel free to reopen it.