[inline_inbuilt_nn_modules] Long compilation time for hf_T5_generate inference cause timeout

[shunting314]

🐛 Describe the bug

The pass status for hf_T5_generate inference has been flaky in a few settings for the past month link

The root cause is hf_T5_generate takes too long to compile. Here summarizes it's compilation time in different settings (dashboard link ):

• default: 223s
• cudagraphs: 1669s
• inductor_max_autotune: 1845s
• cudagraphs_freezing: 549s
• inductor_with_cudagraphs_freezing_autotune: 770s

I've seen timeout in both cudagraphs and inductor_max_autotune settings. Max autotune do increase compilation time but I think the main issue is not max autotune here.

Error logs

..

Minified repro

..

Versions

..

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @oulgen @jamesjwu @aorenste @laithsakka @bdhirsh

[shunting314]

cc @anijain2305

[anijain2305]

This should be fixed now. We had a recent regression fixed by this revert - #121864

Can you please confirm and close the issue if the issue is fixed?

[shunting314]

@anijain2305 a update here. The issue still exists.

In the latest dashboard run for inference link

hf_T5_generate fail in cudagraphs and cudagraphs_freezing setting due to timeout.

[anijain2305]

Seems like there are recompilations.

[anijain2305]

I am seeing excessive recompilations here. Definitely something bad. I remember seeing similar pattern in user empath day

https://gist.github.com/anijain2305/4e2639684bbc13d28ef6de1fb5efc2b4

Turning high priority on this as this one is easier to repro then User empathy day.

TORCH_LOGS=recompiles python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --backend eager --disable-cudagraphs --device cuda --only=hf_T5_generate