Add global gradcheck setting

[albanD]

🚀 Feature

There as two main use cases for gradcheck:

• User facing API to help them check their custom autograd
• Our own testing that use it to check that all our submodules work well with autograd

The problem that we encounter today while adding new features such as vmap and forward AD is that while we want the user facing API to remain stable to avoid breaking user code, we would like the internal testing to test these by default to make sure that functions we provide work with all these gradient-computing features.

Motivation

We want to avoid having to modify each call to gradcheck in our test suite when we add a new feature.
We also want to make sure new test writer get this by default and it is hard for them to forget to test these.

Pitch

Provide a global config for gradcheck that allows to change the default to the gradcheck and gradgradcheck functions.
It will consist in:

• Add a config to gradcheck submodule with all the relevant settings.
• Add API to set these settings easily for test. Also a decorator-based setting to disable them (to easily disable some settings for a single test).
• Changing the API of gradcheck and gradgradcheck to take None as default that will translate to whatever is currently in the global config if no value is passed.
• Update our default Test class to set the right setting on setUp and restore the current state in onExit.

Alternatives

Alternative would be to change all the calls we have in our test suite, but that wouldn't cover new tests being written.
Changing the default of all gradcheck is possible as well but would lead to BC-breaking behavior for users which is not great.

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved

[gchanan]

How much is gradcheck actually used externally? I'm not sure I buy that breaking BC here regularly isn't the right choice (if it's not used much, if it adds additional checks with clear instructions on opting out).

[albanD]

@gchanan a quick github search seems to indicate quite a lot: https://github.com/search?l=Python&q=torch.autograd.gradcheck&type=Code
There are few false positive in there but from checking some pages randomly almost all of them are actually using our version of gradcheck.

[albanD]

From discussing with @gchanan offline, he thinks that having a custom function that should be used inside out tests is actually a feasible thing and that would be cleaner than a global config.
In particular, he argues that most new tests are written from copy/paste of other code and that we could also prevent use of autograd.gradcheck using a lint rule.
@zou3519 what do you think?

[zou3519]

A custom function and lint rule sounds reasonable to me, it should be less complex than maintaining a global config and the lint rule (if that is possible) will steer new test writers away from using autograd.gradcheck

[mruberry]

I agree that a new function sounds best. We could even put it under testing/_internal if we think the API will change frequently, or torch.testing if we think the API will be stable.

[zou3519]

We could even put it under testing/_internal if we think the API will change frequently, or torch.testing if we think the API will be stable.

The API will probably change frequently (e.g. we will add various different testing options), so testing/_internal sounds like a better place to be.

A simplified version of this would be helpful for cleaning up batched grad testing so I think I will toss something into testing/_internal, minus the lint rule, for now.

[zou3519]

@soulitzer should we keep this issue open until we have a lint rule (or should that be a separate issue?)

[soulitzer]

It probably makes sense for lint rule should be part of this same issue - reopening.