Remove usage of AutoNonVariableTypeMode

[robogast]

I get the following warning when running my model (which uses transposedconvolutions):

[W LegacyTypeDispatch.h:74] Warning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (function operator()) 

[jgong5]

@robogast Can you provide a test case that produced this warning?

[robogast]

AFAIK, it's just using any module which uses AutoNonVariableTypeMode:
(in the example I'm just using 1 as dummy dimenion everywhere, in my real code the dimensions are obviously larger)

>>> import intel_extension_for_pytorch as ipex
>>> import torch
>>> from torch.nn import ConvTranspose2d
>>> from torch.optim import Adam
>>>
>>> model = ConvTranspose2d(1, 1, 1)
>>> optimizer = Adam(model.parameters())
>>>
>>> model_ipex, optimizer_ipex = ipex.optimize(model, optimizer=optimizer)
>>>
>>> model_ipex(torch.randn(1,1,1,1))
[W LegacyTypeDispatch.h:74] Warning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (function operator())
tensor([[[[-0.9811]]]], grad_fn=<IPEXConvTransposeOp>>)

A quick search in GitHub reveals 5 of such modules:

[jgong5]

Thanks. Seems it is something we should avoid using in IPEX. It is recommended to replace with InferenceMode. cc @jiayisunx

[jgong5]

@robogast Just to make sure - you were not encountering functional or performance issues with it, right?

[robogast]

@jgong5 actually I'm not sure: the reason I'm using TransposedConvolutions in the first place is because ATen Bicubic interpolation backwards implementation only runs on one core, meaning I was severely bottlenecked by it (checkout this horrible trace...)

After replacing my bicubic layers with transposedconvs, my sec/it dropped from 18 to 3 (!)

So far, I'm only doing benchmarking, i.e. I only care about CPU utilization and sec/it, but I do see my loss dropping very slowly when I'm using transposedconvs, much slower than with bicubic layers.

However, I couldn't say for sure whether that's because of a fault in transposedconv or because transposedconvs are just inherently worse than bicubic upsampling in my situation...

[alexsin368]

what torch version are you using? i'm guessing it's before 1.10