[inductor][cpu] [amp] acc check fails

[yudongsi]

amp cpu accuracy check failed models

Repro

bash inductor_single_test.sh
single inference performance SUITE MODEL amp first static default 0

new_failures

name	accuracy	perf	suite	reason(reference only)
*	*	*	*	*
gluon_inception_v3	X	√	timm_models	gluon_inception_v3, [2023-08-26 08:23:43 745] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.04454 (ref-fp64): 0.01602 and shape=torch.Size([1 1000])
lcnet_050	X	√	timm_models	lcnet_050, [2023-08-26 08:28:13 614] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.26560 (ref-fp64): 0.03031 and shape=torch.Size([1 1000])
mobilenetv2_100	X	√	timm_models	mobilenetv2_100, [2023-08-26 08:31:40 492] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.03747 (ref-fp64): 0.01834 and shape=torch.Size([1 1000])
mobilevit_s	X	√	timm_models	mobilevit_s, [2023-08-26 08:33:13 423] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 6.00970 (ref-fp64): 2.72817 and shape=torch.Size([1 1000])
mobilenetv3_large_100	X	√	timm_models	mobilenetv3_large_100, [2023-09-02 09:18:52 786] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.01714 (ref-fp64): 0.00836 and shape=torch.Size([1 1000])
LayoutLMForSequenceClassification	X	√	timm_models	LayoutLMForSequenceClassification, [2023-09-09 07:26:27 469] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.00157 (ref-fp64): 0.00046 and shape=torch.Size([1 2])
*	*	*	*	*

SW info

SW	Nightly commit	Main commit
Pytorch	91bde86	e00bd83
Torchbench	/	770d5cf7
torchaudio	dc83b38	66f661d
torchtext	c11d758	60bea66
torchvision	58366ab	a6dea86
torchdata	1d231d1	757c032
dynamo_benchmarks	f228c8b	/

[XiaobingSuper]

Those models failed because of conv+bn folding, it is a big gap compared with the eager model, we need to update using a high tolerance if the real data set gets similar accuracy.

[leslie-fang-intel]

I have picked one of above models mobilenetv2_100 with cmd bash inductor_single_test.sh multiple inference accuracy timm_models mobilenetv2_100 amp first static default 0 on CPX system.
Here is the test log

Testing with freezing on.
multi-threads testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  mobilenetv2_100
skipping cudagraphs for unknown reason
pass
dev,name,batch_size,accuracy,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks

@ESI-SYD May I know is there anything missing? Is this issue platform related?

[yudongsi]

mobilenetv2_100 only failed on single core scenery, pls try bash inductor_single_test.sh single inference accuracy timm_models mobilenetv2_100 amp first static default 0. Also I transfer mobilenetv3_large_100 acc issue from #107367 and missed failed model LayoutLMForSequenceClassification

[leslie-fang-intel]

Thanks @ESI-SYD. I have tried above 6 models' and here are the status on my local CPX system.

• For gluon_inception_v3, lcnet_050, mobilevit_s, I can reproduce the failures and work in progress to debug. Are these 3 models' accuracy issue regression? If so, could you share the last PyTorch commit which can pass the accuracy check for these 3 models?
• For LayoutLMForSequenceClassification, I can't run it with the CMD you provided: bash inductor_single_test.sh single inference accuracy timm_models LayoutLMForSequenceClassification amp first static default 0. Could you help to check if anything I am missing?

loading model: 0it [00:00, ?it/s]Failed to load model: Unknown model (LayoutLMForSequenceClassification). Trying again (1/5) after 30s

• For mobilenetv2_100 and mobilenetv3_large_100, single core scenery also pass in my local CPX system:

(inductor) [root@CPX-4 pytorch]# bash inductor_single_test.sh single inference accuracy timm_models mobilenetv2_100 amp first static default 0
Testing with freezing on.
single-thread testing....
loading model: 0it [00:00, ?it/s]
cpu  eval  mobilenetv2_100
skipping cudagraphs for unknown reason
pass
dev,name,batch_size,accuracy,calls_captured,unique_graphs,graph_breaks,unique_graph_breaks
cpu,mobilenetv2_100,1,pass,250,1,0,0

Here is my software version:

[conda] bert-pytorch              0.0.1a4                   dev_0    <develop>
[conda] clip-anytorch             2.5.2                    pypi_0    pypi
[conda] coca-pytorch              0.0.12                   pypi_0    pypi
[conda] dalle2-pytorch            1.14.2                   pypi_0    pypi
[conda] ema-pytorch               0.2.3                    pypi_0    pypi
[conda] mkl                       2023.2.0            intel_49495    intel
[conda] mkl-include               2023.2.0            intel_49495    intel
[conda] mkl-service               2.4.0           py310hae59892_35    intel
[conda] mkl_fft                   1.3.6           py310h173b8ae_56    intel
[conda] mkl_random                1.2.2           py310h1595b48_76    intel
[conda] mkl_umath                 0.1.1           py310hd987cd3_86    intel
[conda] numpy                     1.25.2                   pypi_0    pypi
[conda] open-clip-torch           2.20.0                   pypi_0    pypi
[conda] pytorch-warmup            0.1.1                    pypi_0    pypi
[conda] rotary-embedding-torch    0.2.7                    pypi_0    pypi
[conda] torch                     2.2.0a0+git18225cc           dev_0    <develop>
[conda] torch-fidelity            0.3.0                    pypi_0    pypi
[conda] torchaudio                2.1.0a0+66f661d          pypi_0    pypi
[conda] torchdata                 0.7.0a0+757c032          pypi_0    pypi
[conda] torchmetrics              1.1.1                    pypi_0    pypi
[conda] torchtext                 0.16.0a0+60bea66          pypi_0    pypi
[conda] torchvision               0.16.0a0+a6dea86          pypi_0    pypi
[conda] vector-quantize-pytorch   1.7.1                    pypi_0    pypi

[leslie-fang-intel]

Update the status:

• 4 of models' failures (gluon_inception_v3, mobilenetv2_100, mobilevit_s and mobilenetv3_large_100) will be fixed by Inductor: Increase multiplier to 3 for Inductor AMP benchmark correctness check #109097
• LayoutLMForSequenceClassification and lcnet_050 need further investigation.

[leslie-fang-intel]

• 4 of models' failures (gluon_inception_v3, mobilenetv2_100, mobilevit_s and mobilenetv3_large_100) has be fixed by Inductor: Increase multiplier to 3 for Inductor AMP benchmark correctness check #109097
• LayoutLMForSequenceClassification should be false alarm which pass in CPX and with little gap to the pass criteria in SPR.
• lcnet_050 tried in model level accuracy testing, the accuracy loss exists after doing conv_bn folding with float32 data type and then convert the weights to bfloat16. May be we need to disable conv_bn folding for this model with AMP.