Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
[Dynet-92]. Multi-device support #704
To reviewer @neubig , you can have a quick test using code below:
changed the title from
Dynet 92 model parallelism
Dynet 92 Multi-device support
Jul 17, 2017
In general, this is great: I think multi-device support will be a great feature for DyNet to have. First, I have a high level comment. In my mind, there are two design decisions here:
How do we specify the "default" device of a graph node when it is not specified explicitly
The first has the advantage of perhaps being easier to understand, but may result in hidden memory moves where people aren't expecting them. It also adds some code complexity.
When some of the inputs are not on the same device what do you do?
My opinion: I tend to prefer 2./2. respectively, but could be convinced otherwise.
This is really great!
I like options 2 and 2 also, but would like to propose a variation of the second (2):
Another proposal (maybe its already there, I didn't look at the code) is to also allow multiple CPU devices. There, the copying will be NOPs, but we can still run different cpu devices as different threads.
How do things look in terms of synchronization in the current implementation?
Very helpful comments!! I also prefer 2. / 2. then. And I think to support both copy like
Currently, we don't support specifying CPU id. I will think about that in the near future.
The remark of
This pull request is review-ready. It will not affect old code and interface and I will split remaining work in the future pull requests.
The remaining things include,
Thanks, this is great! I have a bunch of small comments, but I think once they're resolved and I can confirm that this works in my environment, I think we can merge. Also, some of my comments might just be oversights, so if there's anything that you don't think needs to be fixed, just tell me.