torch.cat without copying memory

[defoishugo]

🚀 The feature, motivation and pitch

Principle

Today, the concatenation implemented on pytorch consists in the allocation of a new tensor.
I would like to know if it is possible to realize a concatenation of contiguous and/or non-contiguous tensors without memory duplication.

Example 1 : contiguous concatenation

The following code is executed with allocation of a new tensor concatenated_tensor:

import torch 

tensor1 = torch.rand(2,3,4)
tensor2 = torch.rand(2,3,4)
concatenated_tensor = torch.cat(tensor1, tensor2)

I'd like to enable the same scenario, but have concatenated_tensor as a view of tensor1 and tensor2.
In terms of UX, I don't know what to propose.

Note: since I'm a new pytorch user, maybe the word "view" is not appropriate. The low-level idea is to consider concatenated_tensor as a list of pointers to tensors.

Example 2 : non-contiguous concatenation

Next, I would like to enable the following scenario, if possible:

import torch 

tensor1 = torch.rand(2,3,4).expand(10, -1, -1)
tensor2 = torch.rand(2,3,4).expand(10, -1, -1)
concatenated_tensor = torch.cat(tensor1, tensor2)

Alternatives

No response

Additional context

Discussed in #70283 with @ejguan.
See discussion/34609.

cc @mruberry @rgommers

[rgommers]

I would like to know if it is possible to realize a concatenation of contiguous and/or non-contiguous tensors without memory duplication.

The answers of @albanD on https://discuss.pytorch.org/t/concatenate-tensors-without-memory-copying/34609/13 explain how difficult this is. It does not fit the strided model of a tensor at all. It'd be a ton of work - I think it's safe to say this won't happen.

I'll let someone else decide, but I propose to close this issue.

[defoishugo]

No problem, we'll close this issue.

[ngimel]

nestedtensor https://github.com/pytorch/nestedtensor might be doing what you want. Also, if you don't need autograd, _foreach_* ops support operations on the lists of tensors, so you don't need to concatenate them in advance.