[inductor][cpu]mobilenet_v3_large PTQ/QAT performance regression in 2024-05-04 nightly release

[zxd1997066]

🐛 Describe the bug

mobilenet_v3_large PTQ/QAT performance regression

model_name	ptq_new	ptq_cpp_new	qat_new	ptq_old	ptq_cpp_old	qat_old	ptq ratio(new/old)	ptq_cpp ratio(new/old)	qat ratio(new/old)
mobilenet_v3_large-eval_throughput	98.34567144961288	98.31156696423297	101.19172882398914	5636.598043580855	5962.843935557124	6856.838217616331	0.02	0.02	0.01

SW info

SW	Branch	Target commit	Refer commit
Pytorch	nightly	1b7523f	02b1ebb
Torchbench	chuanqiw/inductor_quant	ee35d764	ee35d764
torchaudio	nightly	ea437b3	ea437b3
torchtext	nightly	b0ebddc	b0ebddc
torchvision	nightly	06ad737	2c4665f
torchdata	nightly	11bb5b8	0790338
dynamo_benchmarks	nightly	nightly	nightly

Repro:

git clone -b chuanqiw/inductor_quant https://github.com/pytorch/benchmark.git
cd benchmark
pip install --no-deps -r requirements.txt
pip install --no-cache Jinja2==3.1.2 markupsafe==2.0.1 beartype==0.15.0 && pip install mpmath==1.3.0
python install.py --continue_on_fail
export LD_PRELOAD=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}/lib/libiomp5.so:${CONDA_PREFIX:-"$(dirname $(which conda))/../"}/lib/libjemalloc.so
export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:-1,muzzy_decay_ms:-1"
#QAT
TORCHINDUCTOR_FREEZING=1 python run_benchmark.py cpu -m mobilenet_v3_large --torchdynamo inductor --quantize --is_qat --launcher --launcher-args="--throughput-mode" -b 128 --metrics throughputs
mv .userbenchmark/cpu qat
cat qat/metric* # to see the results
#PTQ
TORCHINDUCTOR_FREEZING=1 python run_benchmark.py cpu -m mobilenet_v3_large --torchdynamo inductor --quantize --launcher --launcher-args="--throughput-mode" -b 128 --metrics throughputs
mv .userbenchmark/cpu ptq
cat ptq/metric* # to see the results

Suspected guilty commit: e592a60
torchbench-mobilenet_v3_large-inference-qat-performance-drop_guilty_commit.log
cc @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang @WeizhuoZhang-intel @chuanqi129