[inductor] Triton Auto Tuning - Long Compilation Time

[anijain2305]

Repro -
rm -rf /tmp/torchinductor_$USER; python benchmarks/dynamo/timm_models.py --training --performance --device cuda --inductor --float32 --only=coat_lite_mini

Skipping autotuning leads to 25 seconds of compilation time vs 360 seconds.

Questions

• Why is autotuning so expensive for this model? Can we do generate better Triton code to avoid excessive autotuning?
• It seems that we are not doing parallel autotuning. It could be because parallel autotuning can affect the autotuned results. Just pointing this out for confirmation - https://github.com/pytorch/pytorch/blob/master/torch/_inductor/triton_ops/autotune.py#L64

cc @jansel @ngimel @zdevito