`backward` implicit conversion from tuple to `torch.Tensor` results in an indexing error message

[pbelcak]

🐛 The error message bug

Writing a custom backward pass by subclassing torch.autograd.Function and returning (torch.Tensor,) instead of the expected torch.Tensor as a gradient output per element received results in

TypeError: only integer tensors of a single element can be converted to an index

reported with the call stack to the backward() function.

This was also previously reported in torchdiffeq.

The issue is not that returning (torch.Tensor,) does not work, but that the error message is grossly misleading. I believe this error message can be tracked all the way to the implicit tensor type conversion.

Expected behaviour
It would seem that it would save a few people hours of debugging time if the TypeError was worded, in this specific case, differently:

TypeError: cannot convert an object of type tuple to torch.Tensor

Example code

import torch
from torch.autograd import Function

class MyFunction(Function):
	@staticmethod
	def forward(ctx, input: torch.Tensor):
		ctx.save_for_backward(input)
		return input
	
	@staticmethod
	def backward(ctx, grad_output):
		input = ctx.saved_tensors # returns (torch.Tensor,) and not torch.Tensor!
		return input * grad_output # causes the TypeError in backward pass stemming from loss.backward()

Versions

[pip3] numpy==1.26.2
[pip3] pytorch-msssim==1.0.0
[pip3] torch==2.1.2
[pip3] torchaudio==2.1.1
[pip3] torchvision==0.16.2
[pip3] triton==2.1.0
[conda] numpy                     1.26.2                   pypi_0    pypi
[conda] pytorch-msssim            1.0.0                    pypi_0    pypi
[conda] torch                     2.1.2                    pypi_0    pypi
[conda] torchaudio                2.1.1                    pypi_0    pypi
[conda] torchvision               0.16.2                   pypi_0    pypi
[conda] triton                    2.1.0                    pypi_0    pypi

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @lezcano @Varal7

[albanD]

Hey!

Running this code raises the following error for me:

  File "/home/albandes/local/pytorch/3.11_debug_source/torch/autograd/graph.py", line 681, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/albandes/local/pytorch/3.11_debug_source/torch/autograd/function.py", line 289, in apply
    return user_fn(self, *args)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/albandes/local/pytorch/3.11_debug_source/test/foo.py", line 13, in backward
    return input * grad_output # causes the TypeError in backward pass stemming from loss.backward()
           ~~~~~~^~~~~~~~~~~~~
TypeError: only integer tensors of a single element can be converted to an index

Which points directly to the line in your custom Function where the error happens.

Note that doing (torch.rand(2),) * torch.rand(2) will through the exact same error so this is quite unrelated to autograd I'm afraid.

[pbelcak]

Hiya! That's what I though -- sorry, I wasn't able to track it down properly. The autograd setting is mentioned mostly to demonstrate how out of place I think the error message is because the conversion actually happens within torch on what is returned by backward, which is code one normally does not look into.

I believe this error message can be tracked all the way to the implicit tensor type conversion.

If you agree that a separate error message would be warranted here, I could try to fix this myself -- sounds like something that should not be too hard.

[albanD]

The problem here is that we're in a weird situation again because our Tensor object is quite tricky :D

To allow scalar Tensors to be used as indices, we implement the __index__ method on the Tensor class.

But when Python does multiplication, the algorithm is roughly:

• Try a.mul(b)
• If NotImplemented try b.rmul(a)
• If NotImplemented
  • If a is a Sequence
    • If b is not an index -> raise "TypeError: can't multiply sequence by non-int of type 'type'" ((torch.rand(2),) * "foo" raises that error)
    • If b is an index, get it's value with .__index__() and repeat the requence that many times
  • If b is a Sequence
    • {same thing with b and a swapped}
  • Else raise the usual unsupported operand type(s) for *:

The problem for you is that, in the context of the __index__ function, this error message makes perfect sense. I do agree it doesn't really make sense from the __mul__ context though.

I guess that the only way to nicely fix that would be to modify the __mul__ and __rmul__ from the Tensor class in python_variable_methods.cpp to pro-actively check if the other element is a sequence. And if so, throw the appropriate error here if this Tensor cannot be converted to an index (with a nice mul-related error) before we get deeper.

Would you be interested in trying to do that? (That's going to be quite a bit of subtle CPython API shenanigans haha)

[pbelcak]

Many thanks for the context! Yes, let me try looking into that. It's low prio so I will take my time though.