Synchronous SGD via layer-wise parallelism

[longjon]

This PR upgrades Caffe with integrated support for layer-wise multi-device parallelism. This allows Caffe to perform parallel synchronous SGD, using model-parallel, data-parallel, or hybrid approaches. This includes parallelization across GPUs, CPU/GPU parallelism, and multi-threaded CPU parallelism. This is a work-in-progress in proof-of-concept state.

The basic approach is as follows:

• Layer parameters are augmented with a device and thread_id. Each provides a (logical) description of which device the layer will be run on (and where its memory will be allocated). Thread IDs work like CUDA streams; layers are launched in their specified order, and automatically synchronized according to the network DAG. Devices and thread IDs are specified manually; poor choices will result in poor performance.
• With one exception, each layer executes on a single device, and has its tops and bottoms and params stored on that same device. The exception is split layer, which is allowed to have tops that live on different devices, and is used to implement automatic copying. The actual top devices have to be specified as a parameter (top_device) to split layer, but this is done automatically for automatically inserted split layers.

Limitations:

• Parameters-as-such may not be shared between devices. Note that Caffe currently provides two different mechanisms for sharing things, which implement essentially the same functionality: parameter sharing and split layers. Rather than extend both versions of this functionality, this PR chooses to extend split layer. Shared parameters can still be used, they just need to be done with Add parameter layer for learning any bottom #2079 and Add option to take parameters from bottoms #2166.
• Each layer's forward and backward must be performed on the same device.
• Automatically inserted split layers are given thread_id -1, and therefore all copies are done in the same thread. This can be worked around by manually specifying split layers.
• There is no way to specify which physical devices to use. (device_ids as given to layers are purely logical and mapped arbitrarily to physical IDs starting from 0.) Use CUDA_VISIBLE_DEVICES instead.
• The currently included examples use DummyDataLayer. Using this with real data layers will cause problems, due to some subtle and unnecessary singleton interaction. Ultimately prefetching should probably just be integrated into this framework, but that requires additional functionality for pipelining computation. There are very ugly ways to hack around this for now if you really want to.
• Many checks that should exist are missing.
• I have only verified correctness of a previous version, there could be bugs in correctness of this one.
• This PR adds a protobuf enum for device types, which should probably be merged with Caffe's own Brew enum somehow.

A simple example is included for CaffeNet data parallelism (not using real data layers). I've found that it performs near 2x on 2 GPUs, ~2.5x on 3 GPUs, and poorly on 4 GPUs. I'm sure that with additional tuning/better execution planning, much nearer-linear scaling can be achieved. The example is generated using #2086 (but the output is included). Note that actually solving will have some (small) additional overhead from H2D transfer of data (which should be made asynchronous) and from the solver parameter-update code.

NVTX instrumentation of forward/backward calls is included, and together with nvprof/nvvp it makes performance issues pretty easy to find. It's a build option, WITH_NVTX.

A hacky option, -notime_layers, is added to caffe time to do rough-and-ready parallel timing.

This PR is largely orthogonal to the net-level parallelism of the parallel branch, even though both can be used to implement some of the same things, like pure data parallelism for synchronous SGD (see #2114). Some common functionality has been factored out. I have no plans to support multi-node parallelism or asynchronous SGD using this code; you can combine it with the parallel branch for that.

There are many existing PRs included in this one:

• Thread-specific singleton (from @cypof) #2067, the thread-specific singleton from @cypof
• Blocking queue (from @cypof, with modifications) #2167, the blocking queue from @cypof, used for thread work queues
• Use CUDA 7's per-thread default stream, with automatic syncing #2077, use CUDA 7 and its per-thread default stream
• Keep track of the layer graph #2073, keep track of the layer graph
• Remove spurious inclusions of net.hpp #2168, remove spurious net.hpp includes so that Net can get intimate with boost::thread
• Add device-parametrized blob accessors #2169, device-parametrized blob accessors, to clean up onerous switches.
• Add parameter layer for learning any bottom #2079, parameter layer and
• Add option to take parameters from bottoms #2166, the param_bottoms option, so that split layer can be used for sharing parameters
• a future PR, which will update cuDNN convolution to be a bit more stringent with its stream synchronization
• in addition, Python net specification #2086 is very useful to avoid writing very long prototxt files

Note that omitting included PRs and (very long) examples, the total diff here is on the order of a few hundred lines.

This is still work-in-progress, but hopefully usable/hackable by eager beavers.

Major TODOs:

• Provide a sensible way to specify (multiple) devices, and make CPU/GPU mode a construction time Net option or coordinate this PR with mode switching some other way.
• Provide a mechanism for pipelining and integrate data loading/transformation into this framework.
• Verify correctness of learning in this framework (any deviation from single-device learning is a bug), and provide tests.
• Verify that known schemes for model/data parallelism perform as expected, e.g., Krizhevsky's one weird trick and data parallelism as employed e.g. by VGG and MSRA.

[futurely]

Why would a PR include and be blocked by so many other PRs?

[dzhwinter]

the peer copy is failed when use cudaDeviceEnablePeerAccess() function in net.cpp line 349.
I test on multiGPU 2 cards

[shelhamer]

Closing as multi-GPU dev took the other turn, although I always liked this branch.