caffe out of memory

[jetyingjia]

@wanglimin
When training use https://github.com/yjxiong/caffe

I find when training, the memory is not a fixed value and is becoming larger, and then this happend.
"""
Out of memory: Kill process 33049 (caffe) score 123 or sacrifice child
Killed process 33049 (caffe) total-vm:118020040kB, anon-rss:32252368kB, file-rss:83732kB
"""
It occures when traing ARTNet-18, but C3D-resnet18 is OK. ALL models and prototxt is official released. Is there something I need to modify the code?
Thank you in advance!

[jetyingjia]

I use this project:
yjxiong/caffe at 3D
https://github.com/yjxiong/caffe/tree/3D

[wanglimin]

I also encounter the problem of out of memory in some of my GPU servers. But in some other GPU servers, this problem did not happen.

I do not know why this problem happens either. It may depends on the OS, libs installed in your server. Also, there may be some bugs in the caffe toolbox.