Misleading Error when doing Large Batch Matrix Multiplication

[askliar]

🐛 Bug

When doing Batch Matrix Multiplication in Pytorch 1.1.0, when too much memory is needed, instead of throwing a meaningful error (the way it was in previous versions e.g. Buy new RAM!),
Pytorch RuntimeError: [enforce fail at CPUAllocator.cpp:56] posix_memalign(&data, gAlignment, nbytes) == 0. 12 vs 0 error is thrown as also mentioned here.

Environment

PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 410.93
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.16.2
[conda] blas 1.0 mkl
[conda] botorch 0.1.0 pypi_0 pypi
[conda] gpytorch 0.3.2 pypi_0 pypi
[conda] mkl 2019.1 144
[conda] mkl-service 1.1.2 py37he904b0f_5
[conda] mkl_fft 1.0.10 py37ha843d7b_0
[conda] mkl_random 1.0.2 py37hd81dba3_0
[conda] pytorch 1.1.0 py3.7_cuda10.0.130_cudnn7.5.1_0 pytorch
[conda] torch-cluster 1.2.4 pypi_0 pypi
[conda] torch-geometric 1.1.2 pypi_0 pypi
[conda] torch-scatter 1.1.2 pypi_0 pypi
[conda] torch-sparse 0.2.4 pypi_0 pypi
[conda] torchvision 0.3.0 py37_cu10.0.130_1 pytorch

[derangedhk417]

I'm not certain that this error always occurs due to a memory issue. I've been having it intermittently on the same data-set. For example, I ran the same exact training process on the same exact data-set 71 times. The only difference between the data-sets was that a single scalar parameter in the objective function was changed. Absolutely none of the tensors changed size. 4 of the 71 runs threw this error, despite the fact that the amount of ram required should not have changed.

[askliar]

@derangedhk417 are you sure you did not have something else occupying your RAM in those 4 runs?

[derangedhk417]

I had them all running sequentially and I didn't do anything else on the system at the time. None of the runs should have been consuming more than half of my ram at any time anyways. My colleague said he's observed memory usage remaining high after he's completely terminated PyTorch in the past. I am wondering if a past run was somehow occupying memory after it terminated.

[littlelittlemoon]

I had them all running sequentially and I didn't do anything else on the system at the time. None of the runs should have been consuming more than half of my ram at any time anyways. My colleague said he's observed memory usage remaining high after he's completely terminated PyTorch in the past. I am wondering if a past run was somehow occupying memory after it terminated.

yes, I have the same problem, and also the memory usage remaining high after I terninated PyTourch, did you have solved this problem?

[zhihanyang2022]

I'm not sure if what I'm going to say is related, but a similar error can be caused by typing this code torch.randn(60000, 1) - torch.randn(60000):

[enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 10000000000 bytes. Error code 12 (Cannot allocate memory)

The reason is that both vectors will be broadcasted as (50000, 50000). So, perhaps your code include some implicit broadcasting?

I created this error once when doing MSE loss over the output vector and the target vector, and using unsqueeze on the first vector solves the problem, if all you want to do is just to compute the element-wise difference.

I think the crux of the issue is that there are just too many items in your matrices, somehow.