Skip to content
Elastic Chainer prototype
Python C++ Makefile
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
echainer
examples
src
tests
.gitignore
LICENSE
Makefile
README.md
echainer.proto
multirunner.py
setup.cfg
setup.py

README.md

Elastic Chainer

Scalable and fault tolerant MPI-like communication library prototype for distributed DNN training in Chainer.

Build

Prerequisites

Runtime

  • protobuf - to run gRPC server
  • grpc - for processes to commumnicate each other
  • cppetcd - to access etcd for distributed coordination
  • glog - for simple logging

Build

  • protoc - protobuf compiler to generate codes
  • g++ - C++ compiler and linkers that support C++14.
  • pkg-config - find all libraries above on compile time

Build (TL;DR)

You'll need python, protoc, and grpc++ for build. Also cppetcd as well.

$ yay -S python protobuf grpc glog
$ which protoc
/usr/bin/protoc
$ which grpc_cpp_plugin
/usr/bin/grpc_cpp_plugin
$ git clone git@github.com:kuenishi/cppetcd.git
$ cd cppetcd && make && make install prefix=/path/to/local && cd ..
$ git clone git@github.com:chainer/echainer.git
$ cd echainer
$ make dev

For build it doesn't need CuPy, but in runtime NcclCommunicator requires it. Also see docker/Dockerfile on how to build cleanly.

Release Package

Build a release binary package (be sure only single platform will be supported by this single build).

$ pip install wheel
$ python setup.py bdist_wheel

Development

Running etcd process in localhost:2379 to run make test is required. To run GPU-less test:

$ make test

To run tests with GPU,

$ make gpu-test

Loglevel

GLOG_logtostderr=1

Copyright

(C) 2018 Preferred Networks, Inc.

You can’t perform that action at this time.