Track the accuracy regress for HF with max-autotune enabled

[shunting314]

🐛 Describe the bug

HF accuracy check starts to regress link

Repro command:

TORCHINDUCTOR_MAX_AUTOTUNE=1 CUDA_VISIBLE_DEVICES=3 python benchmarks/dynamo/huggingface.py --backend inductor --amp --accuracy --only PLBartForCausalLM --training --cold-start-latency

Here are the things I have tried:

• try to build pytorch and run the repro on both the old commit (1b3dc05) and new commit (d8da2a7). Both fail the accuracy check
• try to rollback triton to older pin and run the repro on pytorch commit 1b3dc05. Accuracy check fail
• try to rollback huggingface to the older pin and run the repro on pytorch commit 1b3dc05. Accuracy check pass.

The cause should be huggingface upgrade.
This log may be the reason 'WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine'

Error logs

No response

Minified repro

No response

Versions

x

cc @ezyang @msaroufim @wconstab @bdhirsh @anijain2305

[desertfire]

WARNING:common:fp64 golden ref were not generated for is usually because the fp64 baseline model hits OOM. I think that matches the memory compression ratio drops in those models, so this is likely an ok regression. Maybe relaxing the tol when doing max-autotune?

[shunting314]

Here is the root cause for the regression: https://github.com/huggingface/transformers/pull/24696/files

Previous code like

mask = torch.full((tgt_len, tgt_len), torch.tensor(torch.finfo(dtype).min, device=device), device=device)

works for dtype=float64, but the new code

mask = torch.full((tgt_len, tgt_len), torch.finfo(dtype).min, device=device)

fail because of overflow/underflow

[shunting314]

So I think we know the root cause of the regression. But on the other hand, I'll check if there is any silent issue in max-autotune that cause the problem. Since I think even if we don't have float64 reference generated for accuracy check, ideally we should still pass the check.

Or maybe triton kernel has big numerical difference compared to cublas?

[shunting314]

WARNING:common:fp64 golden ref were not generated for is usually because the fp64 baseline model hits OOM. I think that matches the memory compression ratio drops in those models, so this is likely an ok regression. Maybe relaxing the tol when doing max-autotune?

Hmm, from log:

[2023-09-20 14:09:58,628] torch._dynamo.utils: [WARNING] Similarity score=0.8310787677764893

Current threshold is 0.99, make the check pass may need dramasticly reducing the threshold