New utils.checkpoint rollout.

[albanD]

Thanks to the work done on SavedVariable hooks, we can know have a new implementation of checkpoint that does not require re-entrant autograd and so make it more composable with systems that don't support re-entrant autograd (most notably DDP).

We have a full fledge prototype and testing in #62964 that implement the core logic and ensures it works as expected.

The API we want to introduce here is to have a use_reentrant: bool argument to checkpoint and checkpoint_sequential that control which version of checkpoint is used.

Due to BC concerns we won't be able to just change the default to the new version so we should (one PT version per bullet point):

• Add the new boolean flag that default to True (current behavior) and deprecate not passing the value for it
• Remove the default value for the flag
• Add the a default value of False for the flag

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @albanD @gqchen @pearu @nikitaved @soulitzer @lezcano @Varal7

[albanD]

From discussion the things we absolutely want to test with this new version:

• gradient hooks are properly called once
• it works with autograd.grad
• works when input does require grads but Tensor that require grads are captures (like first layer in a nn)
• works for functions with arbitrary input/output objects
• [optional] distributed specific tests