Merge NVIDIA's NCCL multi-GPU, switch it to python

[cypof]

This is an attempt to merge NVIDIA's multi-GPU work on NCCL, while fixing the open issues with python support and the parallel data pipeline. Some of them are non trivial, particularly around the GIL, so the idea is to switch to a python implementation using processes instead. NCCL supports it, and transfers are direct GPU-GPU anyway, so perf. should be the same. The code in train.py should have the same functionality as the command line and C++ version, but is much simpler and gives more flexibility to users. Custom setups from users will also be easier to share with Caffe2.

To test it, you need to clone NCCL, make install it, and set the new flag in Makefile.config.

The GIL and the fact you need to fork sub-processes before initializing CUDA made it tricky to get good performance, but it looks OK now. It should be as good as the NV branch with the new thread-pool, maybe a bit faster as the current pipeline does an in-device copy as the last step, whereas this one is fully zero-copy using CUDA IPC. I have tried on a couple of 4 and 8-GPU boxes so far and everything seems to stay busy.

If NV is interested to help testing and benchmarking, it seems layer-wise reduction and overlapping NCCL communications with compute is often slower than a single big allreduce at the end. Not sure if I'm doing something wrong, and if it can be fixed/improved. Also I don't think layer-wise works with shared weights. We need to compute actual dependencies between layers and that's better left for Caffe2. For now I simply disable layer-wise if the network has shared weights.

If we are happy with this, I suggest we deprecate multi-GPU from the command line version, and remove all associated code before 1.0. There is a lot of code we could remove around the round-robin IO, shared layers lock, root solvers etc. The complexity has grown a bit out of control, and NV had to add another layer of thread pool in their branch to make it go fast, with more Transformer functions etc. I hope we can avoid it by switching to this.

TODO: breakdown the PR into simpler ones, and modularize the python code. For now it exposes a single function train() that emulates the command line. The different parts could be made easier to use and customize individually.

[cypof]

New commit that allows training with any data layer. Lots of cleaning, and removed most of the old parallel code. I simplified train.py a lot, it's really easy to customize now, and moved the advanced bits like the multi-threaded pipeline to a separate example.

[shelhamer]

This is only a workaround for an Ubuntu 16.04 issue that should be fixed upstream by Ubuntu, or at least that's what I remember. At any rate it should not be committed as part of this patch since it is a build detail and not about parallelism and NCCL.

[shelhamer]

Should have consistent case: next().

[shelhamer]

Should have this->output_labels_ condition.

[jeffdonahue]

Can we avoid most of the InitRand call changes by keeping the argument-less version and invoking InitRand(caffe_rng_rand()) in that case?

[shelhamer]

I did a short pass but I need to review this again with more coffee. In the meantime I made a few points that you could address. I can confirm that this builds and passes tests on multi-GPU machines.

[shelhamer]

I'm a bit confused by the guards here. With the old parallelism removed, shouldn't all of this be guarded?

[cypof]

Yes we could guard the whole file and the corresponding sections in caffe.cpp and python.

[shelhamer]

spellcheck: weights

[shelhamer]

👍 on pulling this into Skip() and Next().

[shelhamer]

Can we take all of these timers back out? I don't think we need to expose the Caffe Timer to Python and the like, since existing profiling tools could be used instead.

[cypof]

Timing is pretty handy to adjust the number of GPUs etc. Not sure how to do that conveniently without it? The only way to measure accurately is to insert events in the GPU stream, so peopl would have to use something like pycuda maybe, is that OK?

[shelhamer]

I've made my peace with this timer. It can stay this time.

[shelhamer]

If this is the same as the proto def, then drop this. No need to have fragile duplication.

[cypof]

I haven't found a way to read the default value from the proto. One way could be to instantiate a data_param, read the value and destroy it. Seems a kill overkill so just copied the constant here.

[shelhamer]

These build checks should be spun out into another PR.

[shelhamer]

Why does the Python target run the checks too?

[cypof]

If people only compile the python target then checks are ignored. Compiling with NCCL but not CUDA is valid I believe but doesn't work at runtime.

[shelhamer]

Is there not a standard or suggested NCCL install path?

[cypof]

If they install it its the default /usr/local path, so they don't need that. I could remove this, not sure if installing would be the typical setup for nccl.

[shelhamer]

This is pretty weird. Let's have a closer look in person at some point.

[shelhamer]

I'd rather see this kind of check in tests than debugging blocks in the main code.

[shelhamer]

Is there some kind of corresponding test we can substitute for these debugging blocks and all the invocations of check_device()? I'd rather do without these.

[shelhamer]

I'm still not so enthused for these, but if they have to stay at least add a comment to document the need for check_device() at its definition.

[jeffdonahue]

All the extra debugging code is a bit ugly, but if it makes debugging easier and doesn't affect performance outside of debug mode (and it looks like it shouldn't as everything's in #ifdef DEBUG guards), I'm ok with it. check_device could use a comment though.

[shelhamer]

Why char* -> string and the need to now call c_str()?

[shelhamer]

Setting the type has no effect when instantiating the solver directly from its type as done here with SGDSolver and the others. It's only needed when making use of the solver registry.

[cypof]

It used to work as the other workers did not apply gradients. Now each worker does the full iteration, so they need to be of the right type, including the ones created by parallel.cpp, based on param type.

[shelhamer]

This should no longer be needed as of #5009. Please drop these set_type0 to check.

[shelhamer]

I'm not sure why this is here, and in particular in this PR.

[cypof]

The python code might want to initialize each process with different log levels. E.g. full log for the master, and warn only for each worker, so that there is less duplicated logs.

[shelhamer]

Logging could be its own commit.

[shelhamer]

It'd be nice, but not strictly necessary, to figure out how to do this with boost python call policies instead of hardcoding it. (Like the Net constructor.)

[shelhamer]

Many of the members exposed in these changes don't seem specific to this PR. Could they be pulled out on their own?

[shelhamer]

How can the sizes differ? This seems like it should never happen.

[cypof]

It's a weird case. As an optimization, Blob doesn't release its buffers if the size is reduced. It only updates its size, and uses part of the allocated buffers. Unfortunately the underlying syncedmem doesn't have a notion of size vs capacity, it always copies its full length between CPU and GPU. If a pointer is directly set from set_cpu_data with the current blob size, syncedmem might try to copy an old larger capacity into it.

[shelhamer]

Right.

[shelhamer]

Shouldn't this reset when seeking back to the beginning or otherwise it will add up endlessly?

[cypof]

It's on purpose, the round robin can continue the same way, even if the database has to reset. It's easier to think of it on the unrolled dataset.

[shelhamer]

At a glance it looks like Next() is still doing the work of loading data. Can Skip() instead advance over the data w/o doing all the work of Next()?

[cypof]

I checked the database code, it's only moving the cursor which should be cheap.

[cypof]

Thanks for reviewing, I will update this tomorrow.

[junshi15]

@cypof since you are getting rid of P2Psync, how would this affect RDMASync used in CaffeOnSpark?

[cypof]

@junshi15 GPUParams has not changed, it still gathers all the weights in a single buffer, so the distributed version should not require any changes. You run the distributed allreduce on this buffer at the end of the local one like before. The code will need to extends NCCL instead of P2PSync, but they are similar.

[junshi15]

@cypof Thanks for the info. How much performance gain do you see with NCCL against original P2PSync?

[cypof]

I don't have numbers yet, but it's better. If you plan to switch I would be happy to know what you get.

[shelhamer]

This shouldn't go missing! Can't drop this from the API.

[shelhamer]

Should not be necessary: please double-check and drop. Restore() should handle this.

[shelhamer]

ImageDataLayer needs a TestSkip() too.

[shelhamer]

multi_process -> multiprocess?

[shelhamer]

@cypof Thanks for the updates. Still need to drop set_iter() and set_type() but this is looking ready otherwise!

[SIshijima]

Although a training for LSTM network with multi-GPU can be completed successfully, I failed to resume the training with the snapshot and multi-GPU.
(Now I'm trying to find the reason, but it looks hard for me.)

Multi GPU training --> Success.
Snapshot by MultiGPU ---> Resume with multi GPU -->Fail.
Snapshot by MultiGPU ---> Resume with single GPU -->Success.

-----Error message when failed ------
I0101 20:55:55.651378 6305 recurrent_layer.cpp:20] Initializing recurrent layer: assuming input batch contains 10 timesteps of 30 independent streams.
I0101 20:55:56.042740 6305 recurrent_layer.cpp:150] Adding parameter 0: W_xc
I0101 20:55:56.042780 6305 recurrent_layer.cpp:150] Adding parameter 1: b_c
I0101 20:55:56.042794 6305 recurrent_layer.cpp:150] Adding parameter 2: W_hc
I0101 20:55:56.043316 6305 recurrent_layer.cpp:20] Initializing recurrent layer: assuming input batch contains 10 timesteps of 30 independent streams.
I0101 20:55:56.462378 6305 recurrent_layer.cpp:150] Adding parameter 0: W_xc
I0101 20:55:56.462421 6305 recurrent_layer.cpp:150] Adding parameter 1: b_c
I0101 20:55:56.462432 6305 recurrent_layer.cpp:150] Adding parameter 2: W_xc_static
I0101 20:55:56.462441 6305 recurrent_layer.cpp:150] Adding parameter 3: W_hc
F0101 20:55:56.544127 6305 solver.cpp:465] Check failed: Caffe::root_solver()

[jeffdonahue]

Unless I'm misunderstanding something, this should just be prefetch_(param.data_param().prefetch()), no? That should just use the default value if there's no explicit data_param or data_param.prefetch set, removing the need to also duplicate and hardcode PREFETCH_COUNT.

[cypof]

You're right thanks, I removed it.

[jeffdonahue]

I couldn't find where this function was called -- does it get called somewhere? If not, it should be removed from this PR.

[cypof]

It's called in parallel.cpp:232, only if USE_NCCL is set.

[jeffdonahue]

Thanks @cypof -- other than the minor comments above, this looks good to me.

[jeffdonahue]

@SIshijima that's strange, not sure why that would be. I've never tried multi-GPU training with recurrent layers -- does that work with Caffe master as is (without this PR)? Also, does single GPU training with recurrence still work with this PR?

[SIshijima]

Thanks @jeffdonahue I'm working Caffe master with this PR, and found the problem.
Caffe master without this PR can not train recurrent network with multi-GPU. (With single-GPU, it works well) #4851

[jeffdonahue]

@SIshijima great, thanks for checking that -- issue is unrelated to this PR then. Feel free to post a separate issue about it if you like. (edit: oops, I now see you already pointed to an existing issue about this.)

[shelhamer]

Thanks @jeffdonahue for the co-review. @cypof this looks ready for merge once you do a pass to groom the history (for instance squashing style fixes like the line edit, and fixes to CMake, etc.).

Since #5154 is an existing issue, it can be resolved by another PR.

[cypof]

OK done, commits look better now

[shelhamer]

@cypof thanks for the migration to NCCL! I think a follow-up PR that updates the interface docs and adds a notebook for Python parallelism would be helpful.

[willyd]

Multi GPU training from python is great, thanks for this work! However, this breaks multi GPU training on Windows as nccl does not build natively on windows.

In case someone is interested to port this change to windows I have started a minimal nccl CMake based build for windows at https://github.com/willyd/nccl/tree/windows. This can build a static library with VS 2015 but I haven't ported any of the tests yet.

Credits to @tbennun for the initial port.

[awan-10]

After this integration of NCCL, will the old multi-GPU code work? or is the NCCL dependency a must to do multi-GPU training from now on?

[cypof]

Yes NCCL became a required dependency for multi-GPU training.

…

On Thu, Jan 19, 2017 at 5:54 AM, Ammar Ahmad Awan ***@***.***> wrote: After this integration of NCCL, will the old multi-GPU code work? or is the NCCL dependency a must to do multi-GPU training from now on? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4563 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4RXpIfPtE9EWk5K1J4Mr2pUS0Z6VZuks5rT2sqgaJpZM4JfmCx> .

[SIshijima]

@shelhamer , I checked the master this PR merged.
Multi-GPU training for LSTMs can be done successfully, but failed to resume training.
-- #5154 occurs still.
I used coco captioning to check.

[happynear]

@shelhamer, @SIshijima , I came across the same problem when I tried to resume training a normal CNN with multi-GPU. I think this line in solver.cpp should be removed because the parameter restoration is applied in every GPU now.

[cypof]

You're right, the check is not valid anymore, removing it.

[oyxhust]

So I can train the python layer on multi-gpu now?

[antran89]

Hi @cypof, @shelhamer,
Is this true that now DataLayer cannot be shared by multi-GPUs in C++ interface? When you skip a data item in database, you advance the cursor into a new position, and we will never visit some items. Is it true? How offset_ variable shared between multiple GPUs? Hope to have you guys explanations. I am confused about this new DataLayer and the fact of deleting DataReader function.

[happyharrycn]

This part is also confusing to me. Currently, each GPU will have its own process and thus its own instance of DataLayer. During training, parameters of layers will be broadcasted using NCCL. However, the DataLayer's db/cursor seems not, leading to asynchronized data fetching orders. Each GPU will have its own order of fetching data and they are not sharing a same cursor.

In theory, it might lead to slower convergence (worst case you may say exactly the same data points on each GPU). Yet with random shuffling / skipping, this should never happen. And this effect will factor out on large-scale datasets. I am curious how this asynchronized data fetching order will play a role on the accuracy/convergence on datasets with different sizes.

[cypof]

It's not mentioned in the PR description, but the current system is exactly equivalent to the old one. Before there was a single instance of the DB, and some code to round-robin items between GPUs, so that training would still be deterministic. Having a single DB cursor seemed to help with the mechanical disks we used when it was written a couple years ago. But it doesn't matter that much, and people use SSDs now on large GPU boxes anyway, so the new implementation is simpler. Each network opens a separate cursor, and skips some items so that they only keep the ones they would have been given by the round robin code. E.g. with 4 solvers, GPU 0 keeps 0, 4, 8, GPU 1 keeps 1, 5, 9, etc. This also allows training from python, where each solver needs its own process.

[happyharrycn]

Thanks @cypof for the explanation! Now I got it.

I wonder how we should capture this information. It is important if someone want to write a new data layer for Multi-GPU setting in C++/Python. They should implement proper round-robin schedule by checking solver_rank.

[iacopomasi]

I thought that with this pull it would be possible to run caffe con MultiGPUs with a custom Python layer but doing so I got:

PythonLayer does not support CLI Multi-GPU, use train.py

I am not sure if I understood correctly: Can I use my custom python layer if I use train.py ?

Thanks

[oyxhust]

I have the same question as @iacopomasi . When I used my custom python layer but got this issue, So I have to use trian.py? @shelhamer

[shelhamer]

@iacopomasi @oyxhust Right, train.py is the demonstration of the new pycaffe multi-GPU interface which includes training nets with Python layers (as each net is now in its own process, sidestepping earlier parallelization issues). For further usage questions please ask on the mailing list.

@cypof it might be helpful to include further documentation in the form of an ipython notebook?

[weiliu89]

@cypof Thanks for including NCCL in Caffe, which helps speed up the Multi-GPU training quite a bit. However, I am facing two issues:

1. As you mentioned that share_weight is not supported for AllReduce, do you have any plan to include support for this? Or can you give any hints on how to add support for it. When a net has share_weight, it can be solved by setting layer_wise_reduce to false. However the training speed becomes much slower (2x). I think supporting share_weight will be very useful (for example recurrent network needs to share_weight).

2. When I cancel a job trained using Multi-GPU, the job seems to hang at ncclCommDestroy. Have you encountered such issue?

Thanks!

[cypof]

@shelhamer @weiliu89 Sorry I missed the previous message. Yes a notebook would be great, I will try to do that. About share_weight, it's a matter of ordering the graph, to sync weights only when all their layers are done. It might not be too difficult but we need to look at the code for weight sharing, how to list layers for a given blob etc. I haven't seen the hang on shutdown, there is not much we can do if it hangs in it. It might be useful to create an issue on the NCCL repo.