CUDA out of memory

[ShuteLee]

python train.py --gpu-id 0 --loss Proxy_Anchor --model r
esnet50 --embedding-size 512 --batch-size 180 --lr 6e-4 --dataset SOP --warm 1 --bn-freeze 0 --lr-decay-step 20 --lr-dec
ay-gamma 0.25
wandb: Currently logged in as: shute (use wandb login --relogin to force relogin)
wandb: wandb version 0.10.10 is available! To upgrade, please run:
wandb: $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.10.8
wandb: Syncing run eager-dream-16
wandb: ⭐️ View project at https://wandb.ai/shute/SOP_ProxyAnchor
wandb: 🚀 View run at https://wandb.ai/shute/SOP_ProxyAnchor/runs/2ca7m47a
wandb: Run data is saved locally in wandb/run-20201113_090754-2ca7m47a
wandb: Run wandb off to turn off syncing.

Random Sampling
Training parameters: {'LOG_DIR': '../logs', 'dataset': 'SOP', 'sz_embedding': 512, 'sz_batch': 180, 'nb_epochs': 60, 'gpu_id': 0, 'nb_workers': 4, 'model': 'resnet50', 'loss': 'Proxy_Anchor', 'optimizer': 'adamw', 'lr': 0.0006, 'weight_decay': 0.0001, 'lr_decay_step': 20, 'lr_decay_gamma': 0.25, 'alpha': 32, 'mrg': 0.1, 'IPC': None, 'warm': 1, 'bn_freeze': 0, 'l2_norm': 1, 'remark': ''}
Training for 60 epochs.
0it [00:00, ?it/s]/home/server8/lst/Proxy-Anchor-CVPR2020-master/code/losses.py:48: UserWarning: This overload of nonzero is deprecated:
nonzero(Tensor input, *, Tensor out)
Consider using one of the following signatures instead:
nonzero(Tensor input, *, bool as_tuple) (Triggered internally at /opt/conda/conda-bld/pytorch_1595629411241/work/torch/csrc/utils/python_arg_parser.cpp:766.)
with_pos_proxies = torch.nonzero(P_one_hot.sum(dim = 0) != 0).squeeze(dim = 1) # The set of positive proxies of data in the batch
Train Epoch: 0 [330/330 (100%)] Loss: 10.849229: : 330it [01:34, 3.49it/s]
Evaluating...
100%|██████████| 337/337 [01:25<00:00, 3.95it/s]
R@1 : 51.770
R@10 : 67.938
R@100 : 81.594
R@1000 : 92.909
0it [00:00, ?it/s]
Traceback (most recent call last):
File "train.py", line 290, in
m = model(x.squeeze().cuda())
File "/home/server8/anaconda3/envs/proj/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/server8/lst/Proxy-Anchor-CVPR2020-master/code/net/resnet.py", line 175, in forward
x = self.model.layer1(x)
File "/home/server8/anaconda3/envs/proj/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/server8/anaconda3/envs/proj/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
input = module(input)
File "/home/server8/anaconda3/envs/proj/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/server8/anaconda3/envs/proj/lib/python3.8/site-packages/torchvision/models/resnet.py", line 112, in forward
out = self.conv3(out)
File "/home/server8/anaconda3/envs/proj/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/server8/anaconda3/envs/proj/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 419, in forward
return self._conv_forward(input, self.weight)
File "/home/server8/anaconda3/envs/proj/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 415, in _conv_forward
return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: CUDA out of memory. Tried to allocate 552.00 MiB (GPU 0; 7.80 GiB total capacity; 6.09 GiB already allocated; 392.69 MiB free; 6.44 GiB reserved in total by PyTorch)

wandb: Waiting for W&B process to finish, PID 4172153
wandb: Program failed with code 1. Press ctrl-c to abort syncing.
wandb:
wandb: Find user logs for this run at: wandb/run-20201113_090754-2ca7m47a/logs/debug.log
wandb: Find internal logs for this run at: wandb/run-20201113_090754-2ca7m47a/logs/debug-internal.log
wandb: Run summary:
wandb: loss 12.3987
wandb: R@1 0.5177
wandb: R@10 0.67938
wandb: R@100 0.81594
wandb: R@1000 0.92909
wandb: _step 0
wandb: _runtime 193
wandb: _timestamp 1605276667
wandb: Run history:
wandb: loss ▁
wandb: R@1 ▁
wandb: R@10 ▁
wandb: R@100 ▁
wandb: R@1000 ▁
wandb: _step ▁
wandb: _runtime ▁
wandb: _timestamp ▁
wandb:
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb:
wandb: Synced eager-dream-16: https://wandb.ai/shute/SOP_ProxyAnchor/runs/2ca7m47a

CUDA out of memory in the second epoch.
I have set the batch size to 30, 100, 150, 180,. Nothing helps.
pytorch 1.6
CUDA 10.1
GPU RTX2080Super 8G

I have spent many hours, but still can not solve.

Many thanks for your help.

[YingjieYin]

I got the same problem, how do you solve it ?

[tjddus9597]

If memory problems occur during training, try reducing the batch size.
If you run into problems during evaluation, especially only on the SOP dataset, set the line 144, "if len(xs)<10000:" of the "evaluate_cos_SOP" function by reducing the constant value.
This line determines the size to compute the cosine similarity.
However, if this number is reduced, the number of loops increases, which may increase the evaluation time.

e.g., utils.py

def evaluate_cos_SOP(X, T, normalize=False):
if normalize:
X = l2_norm(X)

# get predictions by assigning nearest 8 neighbors with cosine
K = 1000
Y = []
xs = []
for x in X:
    if len(xs)<1000:
        xs.append(x)
    else:
        xs.append(x)            
        xs = torch.stack(xs,dim=0)
        cos_sim = F.linear(xs,X)
        y = T[cos_sim.topk(1 + K)[1][:,1:]]
        Y.append(y.float().cpu())
        xs = []
        
# Last Loop
xs = torch.stack(xs,dim=0)
cos_sim = F.linear(xs,X)
y = T[cos_sim.topk(1 + K)[1][:,1:]]
Y.append(y.float().cpu())
Y = torch.cat(Y, dim=0)

# calculate recall @ 1, 2, 4, 8
recall = []
for k in [1, 10, 100, 1000]:
    r_at_k = calc_recall_at_k(T, Y, k)
    recall.append(r_at_k)
    print("R@{} : {:.3f}".format(k, 100 * r_at_k))
return recall

[YingjieYin]

Thank you for your reply, the problem has been solved.