Hook up functorch transforms to the autograd.functional benchmarks

[zou3519]

autograd.functional has benchmarks: https://github.com/pytorch/pytorch/blob/master/benchmarks/functional_autograd_benchmark/utils.py

We should add an option to run them using functorch. It'll probably look something like "if can import functorch, then run them, otherwise, just do the usual thing of autograd.functional"

Motivation

• To get a clearer sense of what functorch perf really looks like when compared with autograd.functional. We have anecdata of order-of-magnitude speedups on various models.

cc @albanD @soulitzer if you folks have thoughts on if the benchmarks should go there or elsewhere

[vfdev-5]

List of issues discovered when running benchmark:

• RuntimeError: bad optional access with multi_head_attention_forward #632
• Cannot access storage of TensorWrapper #629
• Trying to use forward AD with native_dropout that does not support it #628
• Issue with tensor.numpy() for wrapped tensors #626

[vfdev-5]

@zou3519 as pytorch/pytorch#75689 was merged can we close this issue or there is still something else to be done ?

[zou3519]

Nope, that looks good to me