How to use the Graph2Seq model with multiple GPUs?

[nashid]

How to train the Graph2Seq model in a multiple-GPU environment? As an example, there is an NMT example here: https://github.com/graph4ai/graph4nlp/tree/master/examples/pytorch/nmt

The model is built here:
https://github.com/graph4ai/graph4nlp/blob/master/examples/pytorch/nmt/build_model.py

Could this be extended to be trained in a multiple GPU environment?

[AlanSwift]

Yes.

[nashid]

@AlanSwift Can you please point me in the direction of how can I train Graph2Seq in a multiple GPU environment? Do you have any such examples?

[nashid]

@AlanSwift is there a configuration parameter that I have set? Wondering is there an example?

[AlanSwift]

We are sorry that currently we don't have an example for multiple GPUs. You might refer to the pytorch guidance.

[nashid]

@AlanSwift wondering do you plan to add an example for multiple GPUs? That would be really handy.

[AlanSwift]

OK. I will do it in my free time. There is a tip. Since the Graph2Seq model takes the GraphData as input which doesn't support scatter for nn.DataParrel, I think the best choice is nn.parallel.DistributedDataParallel. It is quiet tricky and I will provide a demo soon.

[nashid]

@AlanSwift that would be great. Looking forward to it.

[xiao03]

This issue is closed due to no further request / updates. Please re-open it if you found necessary.

[nashid]

@AlanSwift the issue is closed. But no solution is provided yet.

[nashid]

@AlanSwift wondering do you have any update?

[nashid]

@AlanSwift without the support to run in a multiple-GPU environment, not sure how this library could be useful for large datasets. Would really appreciate it if you have any updates.

[AlanSwift]

@nashid This issue is under discussion now. I will give you solution in around a week. Thanks for your important advice.

[nashid]

@AlanSwift wondering any update? And thanks for looking into it 👍

[AlanSwift]

This part is under discussion now. I can give a temporal solution.
There are two methods for multiple GPUs: 1. nn.DataParallel (DP), 2. nn.DistributedDataParallel (DDP).
The DP solution is here.
The DDP solution is here.

[nashid]

@AlanSwift I will try it out. Wondering when this solution would be part of graph4nlp? Any tentative timeline?