14k github models on PyTorch 2.0 pass rates dashboard #93667
Description
We are weekly running dynamo-eager, dynamo-eager-fullgraph, export and inductor on 14k nn.Modules crawled from 1.4k github projects to get coverage level, find and fix bugs. This dashboard page is used to track the pass rates of different backends.
How to repro:
- Checkout https://github.com/jansel/pytorch-jit-paritybench
- Batch evaluation with different backends:
- dynamo-eager:
python main.py --backend eager - dynamo-eager-fullgraph:
python main.py --backend eager --fullgraph - export:
python main.py --compile_mode export - inductor:
python main.py
- dynamo-eager:
- Adhoc evaluation:
pytest ./generated/{filename}.py -k test_{n}(e.g,pytest ./generated/test_KunpengLi1994_VSRN.py -k test_002)python -e ./generated/{filename}.py --backend eager
Bugs umbrella task(#92670)
cc @ezyang @msaroufim @bdhirsh @anijain2305 @zou3519 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @soumith @wconstab @ngimel @Xia-Weiwen @desertfire @davidberard98