ResourceExhaustedError during retraining in Object detection API

[Garima-Mishra]

I am trying to train the mobilenet v1 with pets data set.
from tensorflow.python.client import device_lib
device_lib.list_local_devices()
gives following output
name: GeForce GTX 750 Ti
major: 5 minor: 0 memoryClockRate (GHz) 1.15
pciBusID 0000:03:00.0
Total memory: 1.95GiB
Free memory: 1.93GiB

But when I run training. I get the error
tensorflow/core/framework/op_kernel.cc:1158] Resource exhausted: OOM when allocating tensor with shape[24,128,75,75]
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>, OOM when allocating tensor with shape[24,128,75,75]

Even tried with pascal dataset and other pre-trained models also. I was able to run call models for running the object detection tutorial.
I followed the instruction given for running_locally.md
One thing I observed that all of the GPU memory get exhausted before throwing the error.
What is the minimum GPU memory required for training?

[jch1]

Hi @Garima-Mishra - most of out models require much more than 2Gb of memory (usually between >10 Gb at training time). A few things you could do (besides get a new GPU):

1. decrease the training resolution
2. decrease the batch size

In each of these cases you may need to experiment a bit with hyperparameters like learning rate.

[Garima-Mishra]

Hi, @jch1
Thanks for the quick response. Can I use multiple GPU's for training as I have two 8GB GPU as well?
I will try with your suggestions of reducing batch size and resolution as well.

[jch1]

Yes, you have to tweak the num_clones parameter (see the object_detection/train.py file).

[SimsGautam]

simply tweaking the num_clones parameter (along with adjusting batch size) doesn't seem to get multi-GPU to work @jch1, see #1782

[tmattio]

I have the same issue running Faster-RCNN with Inception Resnet V2.
I have been running the model on a GeForce GTX 1080 and a Tesla K80 (the GPU on the AWS P2 instances), respectively 8GB and 12GB of memory.

I've tried to change the resolution, down to 400:

image_resizer {
  keep_aspect_ratio_resizer {
    min_dimension: 400
    max_dimension: 400
  }
}

And change the batch_size, down to 8

second_stage_batch_size: 8

But I still have the same error.

[tmattio]

Actually, it's not exactly the same issue. Here is the error:

ResourceExhaustedError (see above for traceback): Ran out of GPU memory when allocating 0 bytes for 
         [[Node: Loss/RPNLoss/Loss_1/SoftmaxCrossEntropyWithLogits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](Loss/RPNLoss/Loss_1/Reshape_2, Loss/RPNLoss/Loss_1/Reshape_3)]]
         [[Node: gradients/FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_11/Conv2d_1x1/convolution_grad/tuple/control_dependency_1/_5227 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_28924_gradients/FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_11/Conv2d_1x1/convolution_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

The Ran out of GPU memory when allocating 0 bytes is interesting...

Any ideas?

[sbmaruf]

I am new to tensorflow and Machine Learning. Recently I am working on a model. My model is like below,

1. Character level Embedding Vector -> Embedding lookup -> LSTM1

2. Word level Embedding Vector->Embedding lookup -> LSTM2

3. [LSTM1+LSTM2] -> single layer MLP-> softmax layer

4. [LSTM1+LSTM2] -> Single layer MLP-> WGAN discriminator

while I'm working on this model I got the following error. I thought My batch is too big. Thus I tried to reduce the batch size from 20 to 10 but it doesn't work.

ResourceExhaustedError (see above for traceback): OOM when allocating
tensor with shape[24760,100] [[Node:
chars/bidirectional_rnn/bw/bw/while/bw/lstm_cell/split =
Split[T=DT_FLOAT, num_split=4,
_device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients_2/Add_3/y,
chars/bidirectional_rnn/bw/bw/while/bw/lstm_cell/BiasAdd)]] [[Node:
bi-lstm/bidirectional_rnn/bw/bw/stack/_167 =
_Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0",
send_device="/job:localhost/replica:0/task:0/device:GPU:0",
send_device_incarnation=1,
tensor_name="edge_636_bi-lstm/bidirectional_rnn/bw/bw/stack",
tensor_type=DT_INT32,
_device="/job:localhost/replica:0/task:0/device:CPU:0"]]

tensor with shape[24760,100] means 247600032/10241024 = 75.*** MB memory. I am running the code on a titan X(11 GB) gpu. What could go wrong. Why this type of error occured?

Extra info: the size of the LSTM1 is 100. for bidirectional LSTM it becomes 200.
The size of the LSTM2 is 300. For Bidirectional LSTM it becomes 600.

Note : The error occurred after 32 epoch. My question is why after 32 epoch there is an error. Why not at the initial epoch.

[MohammadMoradi]

Hi, I have the same problem. I'm training a faster-RCNN for my costume dataset and I got the following error. Batch size is one and I use image_resizer to resize images.
image_resizer {
keep_aspect_ratio_resizer {
min_dimension: 150
max_dimension: 300
}
INFO:tensorflow:global step 4389: loss = 0.1126 (0.149 sec/step) 2018-04-11 12:29:38.587877: E tensorflow/core/common_runtime/bfc_allocator.cc:244] tried to allocate 0 bytes 2018-04-11 12:29:38.587989: W tensorflow/core/common_runtime/allocator_retry.cc:32] Request to allocate 0 bytes 2018-04-11 12:29:38.588018: E tensorflow/core/common_runtime/bfc_allocator.cc:244] tried to allocate 0 bytes 2018-04-11 12:29:38.588037: W tensorflow/core/common_runtime/allocator_retry.cc:32] Request to allocate 0 bytes 2018-04-11 12:29:38.588263: E tensorflow/core/common_runtime/bfc_allocator.cc:378] tried to deallocate nullptr 2018-04-11 12:29:38.588404: E tensorflow/core/common_runtime/bfc_allocator.cc:378] tried to deallocate nullptr 2018-04-11 12:29:38.593826: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: Ran out of GPU memory when allocating 0 bytes for [[Node: Loss/RPNLoss/Loss_1/SoftmaxCrossEntropyWithLogits = SoftmaxCrossEntropyWithLogits[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Loss/RPNLoss/Loss_1/Reshape_2, Loss/RPNLoss/Loss_1/Reshape_3)]]

[sbmaruf]

https://stackoverflow.com/questions/48007984/training-broke-with-resourceexausted-error

follow the answer.

[anonym24]

I have NVIDIA GeForce GTX 1060 6 GB and this error happens to me

It's not about GPU memory

I was training on Notebook with NVIDIA GeForce 940 MX 2 GB and it was working ok

[JigyasaK]

I had the same issue even when my batch_size was already 1. I resized the image to 400x400 in the config file and then it ran without errors.

image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 400
        max_dimension: 400
      }