Skip to content
This repository was archived by the owner on Aug 1, 2025. It is now read-only.
This repository was archived by the owner on Aug 1, 2025. It is now read-only.

[Inductor] [CPU] Accuracy failure in huggingface model XLNetLMHeadModel on WW03.2 #2036

@yudongsi

Description

@yudongsi

🐛 Describe the bug

Compare with the ww02.4, there is a accuracy failure on Huggingface model XLNetLMHeadModel on TorchInductor CPU Performance Dashboard on ww03.2:

WW03.2 SW info:

SW Nightly commit Master/Main commit
Pytorch 868797a d41b5d7
Torchbench / bbdc777
torchaudio ecc2781 4a037b0
torchtext 112d757 c7cc5fc
torchvision ac06efe 35f68a0
torchdata 049fb62 c0934b9
dynamo_benchmarks fac4361 /

WW02.4 SW info:

SW Nightly commit Master/Main commit
Pytorch 466be3f 364f526
Torchbench / bbdc777
torchaudio ecc2781 4a037b0
torchtext 112d757 c7cc5fc
torchvision ac06efe 35f68a0
torchdata 049fb62 c0934b9
dynamo_benchmarks fac4361 /

Error logs

cpu  eval  XLNetLMHeadModel                   [2023-01-16 07:53:30,105] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.01949, (ref-fp64): 0.00000 and shape=torch.Size([])
[2023-01-16 07:53:30,106] torch._dynamo.utils: [ERROR] Accuracy failed for key name loss
FAIL

Minified repro

python -m torch.backends.xeon.run_cpu --core_list 0 --ncores_per_instance 1 benchmarks/dynamo/huggingface.py --accuracy --float32 -dcpu --inductor  --no-skip --dashboard --only XLNetLMHeadModel --batch_size 1 --threads 1


Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions