GPU ran out of memory while training from scratch

[SamihaSara]

I am trying to train TrackRCNN without the pre-trained save model, I did not use the "load" command in "config" files. But I got GPU ran out of memory and then the program got stuck(as attached picture). How can I solve this issue? Should I reduce batch_size and learning-rate? I am using GeForce GTX 1080 Titan GPU with 10GB memory.

[SamihaSara]

I have reduced batch size from 8 to 5, still getting the following message :
" 2020-09-02 21:58:58.883299: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available."
Please suggest how to solve this issue. Thanks.

[ahnonay]

As the message tells you, this warning is nothing to worry about. It will still train. If the program gets killed randomly, you may be running out of RAM, not VRAM

[SamihaSara]

@ahnonay I am trying to train on a GPU server (project is also saved in server) and the execution gets stuck every time. By 'running out of RAM' do you mean its a memory problem with the server or my local pc from which I am accessing the server? Can you give me any suggestions to solve this issue?

[ahnonay]

If you are running out of RAM, that would be the memory on your server. Sorry, I am not able to help you with technical issues regarding your server setup.

[ahnonay]

Closing this for now, please reopen if needed