Scalable and fault tolerant MPI-like communication library prototype for distributed DNN training in Chainer.
- protobuf - to run gRPC server
- grpc - for processes to commumnicate each other
- cppetcd - to access etcd for distributed coordination
- glog - for simple logging
protoc- protobuf compiler to generate codes
g++- C++ compiler and linkers that support C++14.
pkg-config- find all libraries above on compile time
grpc++ for build. Also
cppetcd as well.
$ yay -S python protobuf grpc glog $ which protoc /usr/bin/protoc $ which grpc_cpp_plugin /usr/bin/grpc_cpp_plugin $ git clone firstname.lastname@example.org:kuenishi/cppetcd.git $ cd cppetcd && make && make install prefix=/path/to/local && cd .. $ git clone email@example.com:chainer/echainer.git $ cd echainer $ make dev
For build it doesn't need CuPy, but in runtime
NcclCommunicator requires it. Also see
docker/Dockerfile on how to build cleanly.
Build a release binary package (be sure only single platform will be supported by this single build).
$ pip install wheel $ python setup.py bdist_wheel
Running etcd process in
localhost:2379 to run
make test is required. To run GPU-less test:
$ make test
To run tests with GPU,
$ make gpu-test
(C) 2018 Preferred Networks, Inc.