[feature request] Treat tensor as tuple of tensors in torch.cat

[carlosgmartin]

Issue description

Treat torch.cat(a) as torch.cat(tuple(a)) when a is a tensor, like numpy does.

Code example

import torch
import numpy
a = torch.arange(6).reshape(2,3)
print(numpy.concatenate(a.numpy()))
print(torch.cat(tuple(a)))
print(torch.cat(a))

outputs

array([0, 1, 2, 3, 4, 5])
tensor([0, 1, 2, 3, 4, 5])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cat(): argument 'tensors' (position 1) must be tuple of Tensors, not Tensor

[soumith]

I'm not sure I understand the reason for the request.
What exactly are you catting in the case of torch.cat(a)?

[carlosgmartin]

In my particular case, I'm creating a graph neural network. Let edges be a tensor of adjacency matrices for each edge type, i.e. a tensor of shape [m, n, n] where m is the number of edge types and n is the number of nodes. Let features be a tensor of feature vectors for each node, i.e. a tensor of shape [m, k] where k is the dimensionality of the feature space. Then edges @ features has shape [m, n, k], and it is the sum of the feature vectors of neighboring nodes for each node and for each edge type. This must be flattened to [n, m * k], i.e. through torch.cat(edges @ features, 1), to get a single vector for each node, which are then passed in parallel through a function to get the new feature vectors for every node in the graph.

In other words, a tensor should be treated as a tensor of tensors when a sequence is expected.

[ssnl]

(edges @ features).transpose(0, 1).reshape(n, -1) ?

[carlosgmartin]

@ssnl That works, though it's less concise. Shouldn't it be possible to treat a tensor like a sequence? Even better would be adding cat as a tensor method (like eig, sum, etc).

[fmassa]

I think the request of making torch.cat support a single tensor as input makes sense.

We already consider a tensor as an iterable (we can do for batch in torch.rand(3, 10) and we will have tensors of size 10).
I believe torch.cat should iterate over any sequence of tensors and concatenate them over the specified dimension.

I'd not vote for adding cat as a method though. Most of the time we use it with tuples / lists, so this pattern wouldn't make sense in this case.

[fmassa]

A simple fix for this issue would be to do something like

def cat(tensors, dim=0)
    return torch.cat(tuple(tensors), dim=dim)

(or using list instead of tuple).

[soumith]

Some of the core devs discussed this today. We dont think this feature request will go in. Here is the reasoning:

• adding torch.cat(tensor, dim) seems prime to silent bugs, where the user forgets to pass the list of tensors and adds a Tensor.
• the cleaner workound torch.cat(tuple(tensors), dim=dim) is concise enough, and captures the operation much more explicitly