V2 Performance Signal Detected by TorchBench CI on '1.13.0.dev20220707+cu113'

[github-actions]

TorchBench CI has detected a performance signal.

Base PyTorch version: 1.13.0.dev20220630+cu113

Base PyTorch commit: 331c0c18033bf4138c9ea468a8c759865dd8ffff

Affected PyTorch version: 1.13.0.dev20220707+cu113

Affected PyTorch commit: 74877943b84cb6d3fc82993ebc8f0e2f8b3c5ce1

Affected Tests:

• test_eval[demucs-cpu-jit]: +4389.03320%
• test_eval[demucs-cpu-eager]: +4322.05495%
• test_train[maml_omniglot-cpu-eager]: +195.90613%
• test_eval[mnasnet1_0-cpu-jit]: +16.48641%
• test_train[pytorch_stargan-cpu-jit]: +2093.05348%
• test_eval[pytorch_stargan-cpu-jit]: +4740.64289%
• test_train[pytorch_stargan-cpu-eager]: +2045.19054%
• test_eval[pytorch_stargan-cpu-eager]: +5137.13835%
• test_train[timm_regnet-cpu-eager]: +19.19378%
• test_train[vision_maskrcnn-cpu-eager]: +368.08349%
• test_eval[vision_maskrcnn-cpu-eager]: +210.62094%

cc @xuzhao9

[xuzhao9]

Root cause commit: CPUBlas: Use opmath_type for alpha/beta scalars

Workflow

[malfet]

@ngimel, @peterbell10 this looks ridiculously slower
Do we run any perf numbers before submitting the change

[ngimel]

Ouch, I'll revert that, but hm, that's very weird, it's supposed to affect only slow fallbacks which we just don't use, and it should have no relation to fp32 at all. Does it accidentally send gemms that should go to mkl to slow fallbacks?
cc @peterbell10

[peterbell10]

Oh, I think I see it. There was a merge conflict where someone has added a BFloat16 overload that no longer matches the function call due to using opmath_t for alpha and beta.

[ngimel]

I don't think torchbench benchmarks bfloat16 on cpu, so these regressions might be fp32, but I'm not sure.

[xuzhao9]

I don't think torchbench benchmarks bfloat16 on cpu, so these regressions might be fp32, but I'm not sure.

That's correct, we don't benchmark bf16. I believe all the regressed tests are fp32. cc @peterbell10

Also, can you please run TorchBench ondemand CI in the revert PR to confirm the performance is restored? It requires a branch on the pytorch/pytorch repo, a forked one won't work. So if you don't have permission, I can help run it instead.

[peterbell10]

In the re-opened PR I've removed the opmath_t template argument entirely to avoid the possibility of the template providing a closer match in argument deduction than the BLAS-accelerated overloads. That seems to have fixed it, as I'm only seeing small swings in the TorchBench results.

[xuzhao9]

@peterbell10 Sorry just want to confirm: if #65839 is to fix the regression, I thought the TorchBench CI result should see big speedup instead of seeing small swings? Or is there another PR that already reverts the regression PR, and #65839 is to confirm we don't regress it again? If it is the later case, seeing small swings is fine to me.

[peterbell10]

The latter. The PR was reverted and I've reopened it. In fact the revert commit is linked in this thread above my first comment.

[xuzhao9]

The latter. The PR was reverted and I've reopened it. In fact the revert commit is linked in this thread above my first comment.

Got it. Sounds good to me then. I will update TorchBench nightly numbers later to confirm the fix.

[xuzhao9]

I can confirm that the regression has been fixed. Closing this issue.