Torchbench models that don't run in dynamo runners #1901
Description
There's small nuances in how the dynamo runners benchmark models that can make certain torchbench models fail
Some models might be explicitly skipped, others might fail because of some dtype conversion. This can be frustrating because if you add a model to torchbench like clip or cm3leon you won't see it in the pt2dashboard so creating this giant tracker issue to solve this
To repro: look at logs in HUD for e.g https://ossci-raw-job-status.s3.amazonaws.com/log/16535270177 and compare to model names in models/ and canary_models/
If something is showing up in unique to torchbench that means it's not showing up in the pt2 dashboard
There are some concrete things we could do better in dynamo runners like starting with loudly erroring but should also track what these failures are
Notably I found this problem out after investigating stable diffusion and cm3leon