Memory efficient implementation of Caffe

[haikuoyao]

Hi,
I saw this caffe implementation which is memory efficient.
https://github.com/Tongcheng/DN_CaffeScript

And I also notice this in wiki

Memory efficient implementation (newly added feature on June 6, 2017)

There is an option -optMemory which is very useful for reducing GPU memory footprint when training a DenseNet. By default, the value is set to 2, which activates the shareGradInput function 
....

Does that caffe use the above memory efficient way to implementation?

Thanks.

[Tongcheng]

Hello @haikuoyao , caffe version does not because Caffe does not support enough computational graph style optimizations for shareMemory. However, my implementation avoids the "2-sidedness" of Caffe, which means within DenseBlock's data memory I didn't use Blob, instead they are pointers. Another issue is Caffe, in multi-gpu case, don't have good load-balance for memory across GPUs.

[haikuoyao]

Thank you, Tongcheng.

It works well on CIFAR data.
While I tried to train on my own dataset, batch size has to be set 4. If I set it bigger, I got a out of memory error.
I also tried https://github.com/liuzhuang13/DenseNetCaffe yesterday. It's the same.
I wonder should I change network to adapt my dataset?
Thanks a million.

[liuzhuang13]

Hello @haikuoyao,

what is your input size? CIFAR image is 32x32. If your image is much larger, you may want to do a downsampling through a conv with stride 2, before feeding the image into the first dense block, to reduce the memory consumption.

[haikuoyao]

Thanks @liuzhuang13 .
yeah, you are right. My images are 224 * 224 which is used for Resnet.
Thanks a lot. It's so helpful.
Gonna close this issue.