How to release gpu memory of intermediate result tensor

[akikaaa]

In the example below, after calling torch.matmul, the gpu memory usage increases by 181796864 bytes, which is almost the sum of the sizes of c and b.transpose(2,3). So I guess the unreferenced intermediate result b.transpose(2,3) is stored in gpu memory. How could I release the gpu memory allocated to this intermediate result to save gpu memory?

import torch
from torch.autograd import Variable
a = Variable(torch.rand(32, 8, 151, 1024), requires_grad=True).cuda()
b = Variable(torch.rand(32, 8, 151, 1024), requires_grad=True).cuda()
torch.cuda.memory_allocated(0) # 316669952
c=torch.matmul(a, b.transpose(2,3))
torch.cuda.memory_allocated(0) # 498466816, increased by 181796864
c.element_size() * c.nelement() # 23348224
b.transpose(2,3).element_size() * b.transpose(2,3).nelement() #158334976

Environment

• PyTorch Version (e.g., 1.0): 1.0.1
• OS (e.g., Linux): centos
• How you installed PyTorch (conda, pip, source): pip
• Build command you used (if compiling from source):
• Python version: 3.6.9
• CUDA/cuDNN version: cuda9.2/cudnn7.4.2
• GPU models and configuration:NVIDIA 1080TI
• Any other relevant information:

cc @ngimel

[ngimel]

The memory would automatically be released if the intermediate result is not used elsewhere (e.g. it is not necessary to store it for backwards pass). However, it won't be reflected in torch.cuda.memory_allocated because that memory would still be managed by pytorch's memory allocator, and would be reported as allocated to cuda driver, but this memory is still available for pytorch process to use.
On a side note, in this case materializing intermediate tensor is not necessary and happens because of inefficient matmul implementation. That can be fixed independently, PRs welcome.

pytorch/aten/src/ATen/native/LinearAlgebra.cpp

Lines 436 to 437 in 77bb41c

	Tensor tensor1_expanded = tensor1.expand(tensor1_expand_size).contiguous().view(tensor1_bmm_view);
	Tensor tensor2_expanded = tensor2.expand(tensor2_expand_size).contiguous().view(tensor2_bmm_view);

[akikaaa]

@ngimel Thanks for your response. But I'm still confused. If the gpu memory used by intermediate result is released but cached, why doesn't the gpu memory usage decrease even after I call torch.cuda.empty_cache()?
Both torch.cuda.memory_allocated(0) and torch.cuda.memory_cached(0) stay the same.

[ngimel]

Your a and b require gradient, intermediate result is retained to be able to compute the gradients and thus is stored for the backward pass. As I've said, if this issue is a bottleneck for you, it is possible (in your particular configuration) to implement this matmul without an intermediate matrix.
BTW, Variable is a deprecated syntax.