Skip to content

C++/MPI proxies for distributed training of deep neural networks.

License

Notifications You must be signed in to change notification settings

spcl/DNN-cpp-proxies

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

62 Commits
 
 
 
 
 
 

Repository files navigation

DNN-cpp-proxies

C++/MPI proxies for distributed training of deep neural networks, including ResNet-50, ResNet-152, BERT-large, CosmoFlow, DLRM, GPT-2, GPT-3, etc. These proxies cover data parallelism, operator parallelism, pipeline parallelism, and hybrid parallelism.

Demo

Compile:

mpicxx gpt2_large.cpp -o gpt2

Run:

mpirun -n 32 ./gpt2

Setup the number of Transformer layers and the number of pipeline stages:

mpirun -n 32 ./gpt2 64 8

About

C++/MPI proxies for distributed training of deep neural networks.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages