Implement first draft of autograd benchmark.

[albanD]

It is quite a lot of code because I pulled some code from torchaudio and torchvision to remove issues I had to get latest version with pytorch built from source while I can't build there libs from source (dependency missing for torchaudio).

The compare script generates table as follows:

model	task	speedup	mean (before)	var (before)	mean (after)	var (after)
resnet18	vjp	1.021151844124464	1.5627719163894653	0.005164200905710459	1.5304011106491089	0.003979875706136227
resnet18	vhp	0.9919114430761606	6.8089728355407715	0.019538333639502525	6.86449670791626	0.014775685034692287
resnet18	jvp	0.9715963084255123	5.720699310302734	0.08197150379419327	5.887938499450684	0.018408503383398056
ppl_simple_reg	vjp	0.9529183269165618	0.000362396240234375	7.526952949810095e-10	0.00038030146970413625	7.726220357939795e-11
ppl_simple_reg	vhp	0.9317708619586977	0.00048058031825348735	5.035701855504726e-10	0.0005157709238119423	3.250243477137538e-11
ppl_simple_reg	jvp	0.8609755877018406	0.00045447348384186625	9.646707044286273e-11	0.0005278587341308594	1.4493808930815533e-10
ppl_simple_reg	hvp	0.9764100147808232	0.0005881547695025802	7.618464747949361e-10	0.0006023645401000977	6.370915461850757e-10
ppl_simple_reg	jacobian	1.0019173715134297	0.0003612995205912739	2.2979899233499523e-11	0.0003606081008911133	1.2609764794835332e-11
ppl_simple_reg	hessian	1.0358429970264393	0.00206911563873291	2.590938796842579e-09	0.0019975185859948397	2.8916853356264482e-09
ppl_robust_reg	vjp	1.0669910916521521	0.0017304659122601151	3.1047047155396967e-09	0.0016218185191974044	4.926861585374809e-09
ppl_robust_reg	vhp	1.0181130455462972	0.0029563189018517733	2.6359153082466946e-08	0.0029037236236035824	1.020585038702393e-08
ppl_robust_reg	jvp	0.9818360373406179	0.0026934861671179533	6.981357714153091e-09	0.00274331565015018	3.589908459389335e-08
ppl_robust_reg	hvp	1.0270848910527002	0.005576515104621649	3.2798087801211295e-08	0.005429458804428577	6.438724398094564e-08
ppl_robust_reg	jacobian	1.0543611284155785	0.00167675013653934	2.3236829349571053e-08	0.001590299652889371	1.2011492245278532e-08
ppl_robust_reg	hessian	1.0535378727082656	0.01643357239663601	1.8450685956850066e-06	0.015598463825881481	2.1876705602608126e-07
wav2letter	vjp	1.0060408105086573	0.3516994118690491	1.4463969819189515e-05	0.349587619304657	9.897866402752697e-05
wav2letter	vhp	0.9873655295086051	1.1196287870407104	0.00474404776468873	1.133955717086792	0.009759620763361454
wav2letter	jvp	0.9741820317882822	0.7888165712356567	0.0017476462526246905	0.8097219467163086	0.0018235758179798722
transfo	vjp	0.9883954031921641	2.8865864276885986	0.008410997688770294	2.9204773902893066	0.006901870481669903
transfo	vhp	1.0111290842971339	8.374398231506348	0.014904373325407505	8.282224655151367	0.04449500888586044
transfo	jvp	1.0080534543381963	6.293097972869873	0.03796082362532616	6.24282169342041	0.010179692879319191

[dr-ci]

💊 CI failures summary and remediations

As of commit 6a685ff (more details on the Dr. CI page):

---

✅ None of the CI failures appear to be your fault 💚

---

• 1/1 broken upstream at merge base 4fc9e95 on Aug 20 from 9:00am to 6:29pm PDT (18 commits; 397325a - 5cf8592)

---

🚧 1 fixed upstream failure:

These were probably caused by upstream breakages that were already fixed.

Please rebase on the viable/strict branch (expand for instructions)

Since your merge base is older than viable/strict, run these commands:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD

Check out the recency history of this "viable master" tracking branch.

• pytorch_linux_backward_compatibility_check_test on Aug 20 from 9:00am to 6:29pm PDT (18 commits; 397325a - 5cf8592)
  • 🔁 rerun

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 68 times.

[zhangguanheng66]

We is a new MultiheadAttention container in torchtext. Are we going to benchmark it?
https://github.com/pytorch/text/blob/bcb9104680eb9dc978a6bbcc2b9ca46cf2bdbed9/torchtext/modules/multiheadattention.py#L5

[albanD]

@zhangguanheng66 We can definitely add it!
What would be a good example of how it is used? Or a simple set of representative random inputs I can use to add it here?

[zhangguanheng66]

@zhangguanheng66 We can definitely add it!
What would be a good example of how it is used? Or a simple set of representative random inputs I can use to add it here?

Here is an example. You will have to add the dependency of torchtext.
https://github.com/pytorch/text/blob/bcb9104680eb9dc978a6bbcc2b9ca46cf2bdbed9/test/data/test_modules.py#L10

[albanD]

@zhangguanheng66 I added the multi head attention as a standalone thing (without downstream net or loss).
I re-used the setting from the tests in torchtext for all the sizes.

[zou3519]

Not done reading yet. Some initial feedback I had was:

• it might be good to add a test that the benchmark script works to prevent bitrot
• Since we've dropped Python 2 support, some type annotations would enhance readability of the code

[zou3519]

Because PyTorch is Python 3+, I'd love to see some type annotations here (and in the other non-model Python files). Type annotations improve the readability of functions, especially with the helper functions in utils.py

e.g.,

def get_wav2letter(device: torch.device) -> Tuple[Callable, Tuple]

[zou3519]

okay I think I read through everything minus torchvision_models and torch_audiomodels

The benchmarking code looks correct; most of my comments are suggestions for readability and structuring

[zou3519]

LGTM, some minor comments

[facebook-github-bot]

@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@albanD merged this pull request in c2511bd.