ONNX Model Dynamic Quantisation Stuck .

[NitinAggarwal1]

Hi Team ,

I am trying to convert a BERT Seq2Seq Model into Intel Neural Compressor ( Its an ONNX Model).

It is stuck at the stage after which no quantisation is happening . I tried the relative accuracy upto 80% . Please guide.

Stuck at this :

2022-11-09 06:56:20 [INFO] Get FP32 model baseline.
Accuracy is : 50.2
mean: 16.99 ms, std: 2.8 ms, min: 12.7 ms, max: 33.83 ms, median: 16.31 ms, 95p: 21.73 ms, 99p: 26.9 ms
2022-11-09 06:56:58 [INFO] Save tuning history to /home/jupyter/spell_projects/nc_workspace/2022-11-09_06-56-07/./history.snapshot.
2022-11-09 06:56:58 [INFO] FP32 baseline is: [Accuracy: 50.2000, Duration (seconds): 38.4878]

No update after this .

YAML Property File

version: 1.0

model: # mandatory. used to specify model specific information.
name: bert
framework: onnxrt_integerops # mandatory. possible values are tensorflow, mxnet, pytorch, pytorch_ipex, onnxrt_integerops and onnxrt_qlinearops.

quantization:
approach: post_training_dynamic_quant # optional. default value is post_training_static_quant.

tuning:
accuracy_criterion:
relative: 0.80 # optional. default value is relative, other value is absolute. this example allows relative accuracy loss: 1%.
exit_policy:
timeout: 0 # optional. tuning timeout (seconds). default value is 0 which means early stop. combine with max_trials field to decide when to exit.
max_trials: 10
random_seed: 5271

def inc_quantize(MODEL_CONFIG,):

ONNX_MODEL_FILE = os.path.join(MODEL_CONFIG['ONNX_MODEL_PATH'],MODEL_CONFIG['ONNX_MODEL_FILE_NAME'])
model = onnx.load(ONNX_MODEL_FILE)

quantizer = Quantization(MODEL_CONFIG['QUANT_CONFIG'])
quantizer.cfg.tuning.accuracy_criterion.relative **= 0.8**
quantizer.model = common.Model(model)
quantizer.tokenizer = transformers.AutoTokenizer.from_pretrained(
    MODEL_CONFIG['BERT_MODEL_TYPE'])
quantizer.eval_func = run_evaluation_benchmark
q_model = quantizer()
QUANTIZED_SAVE_PATH = os.path.join(MODEL_CONFIG['ONNX_MODEL_PATH'],MODEL_CONFIG['ONNX_QUANTIZED_FILE_NAME'])
q_model.save(QUANTIZED_SAVE_PATH)
print("QUANTIZED MODEL Saved at: ",QUANTIZED_SAVE_PATH)

[mengniwang95]

Hi, does 'No update after this' mean no log is printed but it keeps running, or the program exits?

[NitinAggarwal1]

Hi , No log is printed and keeps running. * Does not exit.

If I increase accuracy relative It works with decent accuracy :) .
But if I keep it less (like 0.01 for relative accuracy) . It keeps running , even though in the above case it was able to get the accuracy.

[mengniwang95]

Higher tuning.accuracy_criterion.relative means higher accuracy drop tolerance for the int8 model and is easier to reach. If tuning.accuracy_criterion.relative = 0.01, it will keep running to try different quantization configurations to reach this restrict threshold.

Normally there will be some logs to show the int8 result as below, does your program not show up?

2022-11-09 10:08:32 [INFO] |*Tune Result Statistics|
2022-11-09 10:08:32 [INFO] +--------------------+-----------+---------------+------------------+
2022-11-09 10:08:32 [INFO] | Info Type | Baseline | Tune 1 result | Best tune result |
2022-11-09 10:08:32 [INFO] +--------------------+-----------+---------------+------------------+
2022-11-09 10:08:32 [INFO] | Accuracy | 0.7572 | 0.7570 | 0.7570 |
2022-11-09 10:08:32 [INFO] | Duration (seconds) | 141.8921 | 56.5870 | 56.5870 |
2022-11-09 10:08:32 [INFO] +--------------------+-----------+---------------+------------------+

[NitinAggarwal1]

Thanks @mengniwang95

I have finally got the required output . Issue was with sudo permission . closing this issue !