Skip to content
forked from dmlc/wormhole

Portable, Scalable and Reliable Distributed Machine Learning, support various platforms including Hadoop YARN, MPI, etc.

License

Notifications You must be signed in to change notification settings

SiNZeRo/wormhole

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Wormhole

Portable, scalable and reliable distributed machine learning.

Wormhole is a place where DMLC projects works together to provide scalable and reliable machine learning toolkits that can run on various platforms

Features

  • Portable:
    • Supported platforms: YARN, MPI and Sungrid Engine
    • Planned: docker support
  • Rich support of Data Source
    • All projects can read data from HDFS, S3 or local filesystem
  • Scalable and Reliable

List of Tools

Build

  • copy make/config.mk to root folder
  • modify according to your settings
  • type make or make name-of-tool-you-want

How to Submit Jobs

  • make sure dmlc-core exist in root folder
    • type make dmlc-core to get it
  • Use the submission script in dmlc-core/tracker to submit job to the platform of your choice

Contributing

  • We believe that we can create machine learning tools that are portable and works with each other.
  • Contributing of machine learning projects, tutorials and to core dmlc projects are welcomed.
    • All machine learning projects can depends on dmlc-core, rabit or parameter-server

Project Structure

  • learn contains simple but powerful learning tools in wormhole
  • repo is used to clone other DMLC repos that wormhole can depend on
  • Depending DMLC Libraries
    • dmlc-core gives the core modules of most DMLC projects.
    • rabit provides reliable BSP Allreduce communication.
    • parameter-server provides asynchronize parameter server abstraction.

About

Portable, Scalable and Reliable Distributed Machine Learning, support various platforms including Hadoop YARN, MPI, etc.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • C++ 86.7%
  • Protocol Buffer 4.0%
  • Makefile 4.0%
  • Shell 3.4%
  • Python 1.9%