Output of non-inplace, non-CompositeImplicitAutograd op has TensorImpl > 1 or StorageImpl use_count != 1

[soulitzer]

Codegen checks are being added in #60286 in VariableType to enforce that for non-inplace, non-CompositeImplicitAutograd ops the output has TensorImpl with use_count of 1, and non-view ops have output with StorageImpl use_count of 1.

Notes:

• StorageImpl tests are ordered before TensorImpl tests, so a StorageImpl failure may hide a TensorImpl failure
• If TensorImpl use_count > 1, we are always able to construct a test that would cause StorageImpl to also fail (though this test may not actually exist in the real world).

These functions return tensors with TensorImpl use_count > 1:

`_embedding_bag`, `_embedding_bag_forward_only`,
`lu_unpack`, `_cudnn_rnn_backward`
`q_per_channel_scales`, `q_per_channel_zero_points`,

These 'non-view' functions return tensors with StorageImpl `use_count != 1:

'thnn_conv2d_forward', 'slow_conv3d_forward', 'channel_shuffle',

Functions that return tensors with StorageImpl use_count != 1 (which may also have TensorImpl use_count > 1:

 `dequantize_self`, `cudnn_rnn`

Edit: There were more entries in these lists, but all the convolution backward failures + batch_norm failures are actually due to the use_count being equal to 0 (i.e., the returned tensor is undefined) from the output mask. This should be expected, so in these cases (all cases) we should just relax the test to test for use_count <= 1

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @lezcano @Varal7

[albanD]

Is there a simple answer to why these fail? Like they return some Tensors that are expected to be used somewhere else?

[soulitzer]

• For q_per_channel_scales and q_per_channel_zero_points the scale and zero_point tensors are returned are saved as fields of the Quantizer. This is mostly OK because quantized tensors don't require grad. We should fix the alias information for the JIT though.
• For dequantize_self, the original tensor is returned as-is if it's already a float tensor. This should not be problematic because the backward is not implemented anyway, but we should fix the alias information for the JIT.
• lu_unpack returns a tuple of undefined tensors when unpack_data and unpack_pivots are both false. This is already addressed, so we can just remove from the list.
• channel_shuffle can cause issues because the output may be aliased with input, but there is no view_info Ensure channel_shuffle does not returns a view #61794.
• _embedding_bag and _embedding_bag_forward_only. In _embedding_bag_cpu_impl_out, the statement max_indices = bag_size is executed in certain modes. This can be fixed easily, but its not problematic either because the tensors are integral. (Maybe alias information can be added here as well to indicate that the two outputs are possibly aliased).
• _cudnn_rnn is fine because the output with use_count > 1 is only used internally (only so that it could be saved for backward).
• _cudnn_rnn_backward can sometimes return undefined tensors, this has already been addressed
• thnn_conv2d_forward, slow_conv3d_forward

[albanD]

This is OK because quantized tensors don't require grad.

Note that the alias informations are pretty important for the jit as well.
So if the alias informations are wrong, we should flag these as it might still be a problem, even when autograd is not involved.

[soulitzer]

Good point.
Looking at _cudnn_rnn more closely, currently it takes in as parameter weight_buf an optional tensor which gets unwrapped into a possibly undefined tensor. If the tensor is defined, a new tensor referencing the same TensorImpl is returned; otherwise, a zero-filled tensor is returned.

Discussion of next steps:

• The output doesn't always alias the input and the input is undefined in that case, so not sure if it is possible to mark as a view. It might require some more effort to properly address this
• For the purposes of JIT it seems fine to update the alias info
• From the autograd side, there are no concerns with set_history and overwriting grad_fn since the weight_buf input does not have gradient (this also implies that double backward is not possible since weight_buf is an intermediate output saved for backward, so that is not a concern either).

[albanD]

so not sure if it is possible to mark as a view

For the non-AutogradExplicit functions, it is ok to be conservative with the alias info. For example, things like .contiguous() or .reshape() always mark the input/output to be aliased even though it might not always be true. The same should be used here I think.