Share convolution buffers to reduce memory usage

[shelhamer]

Share the columnation buffer for im2col / col2im transformations across all Caffe convolution layers. The memory usage is now equal to the maximum buffer size instead of the sum over all layers. In particular this is useful for many-layered architectures like the VGG ILSVRC14 19 layer model.

Advice and Cautions:

• This is worth it for fully convolutional models where Caffe convolution is faster than cuDNN.
• No N-D convolution. The buffer shape will be overwritten by other layers.
• No parallelism. Only a single net can do forward / backward at a time. As the buffer is shared by all layers within and across nets, no convolution can be done in parallel. (A fix for parallel nets is to make the buffer a member of net. DAG parallelism is still out in that case, but isn't currently parallelized anyway.)
• This has no effect on cuDNN convolution, but that consumes less memory anyway.

All credit to @longjon who reshaped our world in #594 and suggested this patch in #520 (comment).

master edition of #1291.

Do not merge.

[sguada]

Although this a simple and elegant change, maybe it will be worthy to consider a factory of temporary Blobs which could be shared, re-used across layers, or even across nets.

[shelhamer]

For a more mannered take on sharing buffers see #2009.

[shelhamer]

As a reminder, the static buffer causes this crash on exit:

F0305 21:14:53.266198 1810 syncedmem.cpp:16] Check failed: error == cudaSuccess (29 vs. 0) driver shutting down

this is a known side effect.

[cheer37]

i got the this problem.
CUDA_CHECK(cudaFreeHost(ptr)); in syncedmem.cpp causes crash on exit.
what is the reason of this?
How can i solve this?
Thanks.

[naibaf7]

@shelhamer
In my Caffe branch, I have shared convolution buffers for both CUDA and OpenCL without compromising ND convolution. If you are interested in fixing this soon, I could probably make a PR out of the CUDA part.

[shelhamer]

@naibaf7 that sounds promising, but I won't have a chance to review it until after the ECCV deadline 03/14, and we'll have to check it with the NVIDIA/Flickr parallelism too. Feel free to open the PR when ready all the same. Thanks.

[shelhamer]

Rebased to master @ a1c81ac for use after the great layer reckoning that split headers.

[cheer37]

@shelhamer
how can i solve this side effect?

[naibaf7]

This problem was recently solved for fast inference on the OpenCL branch. No layers are broken.

[aaron-michaux]

This bug prevents other "atexit" handlers from running. It's sloppy to have an error and say "Hey, don't worry about it, it doesn't have side effects!"

Is there a function that can be set to safely shut down caffe? That should be all that's required, right?