Skip to content
Resource scheduling and cluster management for AI
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/ISSUE_TEMPLATE Add troubleshooting guide for jobs (#2302) Mar 13, 2019
build Add error message when image build failed (#2133) Feb 12, 2019
contrib [Protocol Plugin] Add marketplace dropdown in submission form (#2561) Apr 19, 2019
deployment
docs update upgrade doc for v0.12 (#2603) Apr 19, 2019
examples Add Chinese translation (#2344) Mar 21, 2019
marketplace-v2 Job submission v2 (#2260) Apr 15, 2019
marketplace Delete cntk_g2p.yaml from marketplace for it always failed; (#1526) Oct 17, 2018
src Hide task role status (#2619) (#2640) Apr 23, 2019
subprojects/frameworklauncher [Launcher]: Add more info into SummarizedFrameworkInfo (#2435) Mar 28, 2019
version add cluster version info (#2528) Apr 11, 2019
.gitattributes Change the line ending character of js and css file to LF. Mar 7, 2018
.gitignore Diawang/dockercleaner (#2119) Feb 13, 2019
.travis.yml [Protocol Plugin] Replace bootstrap with fabric styles (#2557) Apr 16, 2019
Jenkinsfile Job submission v2 (#2260) Apr 15, 2019
LICENSE Initial commit Sep 25, 2017
README.md
README_zh_CN.md update upgrade doc for v0.12 (#2603) Apr 19, 2019
paictl.py [Job Debugging] Basic Implement Of Job Debugging. (#2272) Mar 7, 2019
pailogo.jpg add logo Apr 18, 2018
sysarch.png

README.md

Open Platform for AI (OpenPAI) alt text

Build Status Coverage Status Join the chat at https://gitter.im/Microsoft/pai Version

简体中文

OpenPAI is an open source platform that provides complete AI model training and resource management capabilities, it is easy to extend and supports on-premise, cloud and hybrid environments in various scale.

Table of Contents

  1. When to consider OpenPAI
  2. Why choose OpenPAI
  3. Get started
  4. Deploy OpenPAI
  5. Train models
  6. Administration
  7. Reference
  8. Get involved
  9. How to contribute

When to consider OpenPAI

  1. When your organization needs to share powerful AI computing resources (GPU/FPGA farm, etc.) among teams.
  2. When your organization needs to share and reuse common AI assets like Model, Data, Environment, etc.
  3. When your organization needs an easy IT ops platform for AI.
  4. When you want to run a complete training pipeline in one place.

Why choose OpenPAI

The platform incorporates the mature design that has a proven track record in Microsoft's large-scale production environment.

Support on-premises and easy to deploy

OpenPAI is a full stack solution. OpenPAI not only supports on-premises, hybrid, or public Cloud deployment but also supports single-box deployment for trial users.

Support popular AI frameworks and heterogeneous hardware

Pre-built docker for popular AI frameworks. Easy to include heterogeneous hardware. Support Distributed training, such as distributed TensorFlow.

Most complete solution and easy to extend

OpenPAI is a most complete solution for deep learning, support virtual cluster, compatible Hadoop / Kubernetes eco-system, complete training pipeline at one cluster etc. OpenPAI is architected in a modular way: different module can be plugged in as appropriate.

Related Projects

Targeting at openness and advancing state-of-art technology, Microsoft Research (MSR) had also released few other open source projects.

  • NNI : An open source AutoML toolkit for neural architecture search and hyper-parameter tuning. We encourage researchers and students leverage these projects to accelerate the AI development and research.
  • MMdnn : A comprehensive, cross-framework solution to convert, visualize and diagnose deep neural network models. The "MM" in MMdnn stands for model management and "dnn" is an acronym for deep neural network.

Get started

OpenPAI manages computing resources and optimizing for machine learning. Through docker technology, the computing hardware are decoupled with software, so that it's easy to run distributed computing, switch with different deep learning frameworks, or run jobs on consistent environments.

As OpenPAI is a platform, deploy a cluster is first step before using. A single server is also supported to deploy OpenPAI and manage its resource.

If the cluster is ready, learn from train models about how to use it.

Deploy OpenPAI

Follow this part to check prerequisites, deploy and validate an OpenPAI cluster. More servers can be added as needed after initial deployed.

It's highly recommended to try OpenPAI on server(s), which has no usage and service. Refer to here for hardware specification.

Prerequisites and preparation

  • Ubuntu 16.04 (18.04 should work, but not fully tested.)
  • Assign each server a static IP address, and make sure servers can communicate each other.
  • Server can access internet, especially need to have access to the docker hub registry service or its mirror. Deployment process will pull Docker images of OpenPAI.
  • SSH service is enabled and share the same username/password and have sudo privilege.
  • NTP service is enabled.
  • Recommend not to install docker or docker's version must be higher than 1.26.
  • OpenPAI reserves memory and CPU for service running, so make sure there are enough resource to run machine learning jobs. Check hardware requirements for details.
  • Dedicated servers for OpenPAI. OpenPAI manages all CPU, memory and GPU resources of servers. If there is any other workload, it may cause unknown problem due to insufficient resource.

Deploy

The Deploy with default configuration part is minimum steps to deploy an OpenPAI cluster, and it's suitable for most small and middle size clusters within 50 servers. Base on the default configuration, the customized deployment can optimize the cluster for different hardware environments and use scenarios.

Deploy with default configuration

For a small or medium size cluster, which is less than 50 servers, it's recommended to deploy with default configuration. if there is only one powerful server, refer to deploy OpenPAI as a single box.

For a large size cluster, this section is still needed to generate default configuration, then customize the deployment.

Customize deployment

As various hardware environments and different use scenarios, default configuration of OpenPAI may need to be optimized. Following Customize deployment part to learn more details.

Validate deployment

After deployment, it's recommended to validate key components of OpenPAI in health status. After validation is success, submit a hello-world job and check if it works end-to-end.

Train users before "train models"

The common practice on OpenPAI is to submit job requests, and wait jobs got computing resource and executed. It's different experience with assigning dedicated servers to each one. People may feel computing resource is not in control and the learning curve may be higher than run job on dedicated servers. But shared resource on OpenPAI can improve utilization of resources and save time on maintaining environments.

For administrators of OpenPAI, a successful deployment is first step, the second step is to let users of OpenPAI understand benefits and know how to use it. Users can learn from Train models. But below part of training models is for various scenarios and maybe users doesn't need all of them. So, administrators can create simplified documents as users' actual scenarios.

FAQ

If there is any question during deployment, check here firstly.

If FAQ doesn't resolve it, refer to here to ask question or submit an issue.

Train models

Like all machine learning platforms, OpenPAI is a productive tool. To maximize utilization of resources, it's recommended to submit training jobs and let OpenPAI to allocate resource and run it. If there are too many jobs, some jobs may be queued until enough resource available. This is different with run code on dedicated servers, and it needs a bit more knowledge about how to submit/manage training jobs on OpenPAI.

Note, OpenPAI also supports to allocate dedicated resource besides queuing jobs. Users can use SSH or Jupyter to connect and use like on a physical server, refer to here for details. Though it's not efficient to resources, but it also saves cost on setup and managing environments on physical servers.

Submit training jobs

Follow submitting a hello-world job, and learn more about training models on OpenPAI. It's a very simple job and used to understand OpenPAI job configuration and familiar with Web UI.

OpenPAI VS Code Client

OpenPAI VS Code Client is a friendly, GUI based client tool of OpenPAI. It's an extension of Visual Studio Code. It can submit job, simulate job running locally, manage multiple OpenPAI environments, and so on.

Troubleshooting job failure

Web UI and job log are helpful to analyze job failure, and OpenPAI supports SSH into environment for debugging.

Refer to here for more information about troubleshooting job failure.

Administration

Reference

Get involved

How to contribute

Contributor License Agreement

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Call for contribution

We are working on a set of major features improvement and refactor, anyone who is familiar with the features is encouraged to join the design review and discussion in the corresponding issue ticket.

Who should consider contributing to OpenPAI

  • Folks who want to add support for other ML and DL frameworks
  • Folks who want to make OpenPAI a richer AI platform (e.g. support for more ML pipelines, hyperparameter tuning)
  • Folks who want to write tutorials/blog posts showing how to use OpenPAI to solve AI problems

Contributors

One key purpose of PAI is to support the highly diversified requirements from academia and industry. PAI is completely open: it is under the MIT license. This makes PAI particularly attractive to evaluate various research ideas, which include but not limited to the components.

PAI operates in an open model. It is initially designed and developed by Microsoft Research (MSR) and Microsoft Search Technology Center (STC) platform team. We are glad to have Peking University, Xi'an Jiaotong University, Zhejiang University, and University of Science and Technology of China join us to develop the platform jointly. Contributions from academia and industry are all highly welcome.

You can’t perform that action at this time.