-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove all code besides EDL code; update EDL import path #5
Merged
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
*~ | ||
vendor/ | ||
.glide/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,3 @@ | ||
.Python | ||
*.crt | ||
.cache | ||
vendor | ||
*~ | ||
*.pyc | ||
*.idea | ||
.vscode | ||
vendor/ | ||
.glide/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,20 +1,7 @@ | ||
- repo: https://github.com/dnephin/pre-commit-golang | ||
sha: e4693a4c282b4fc878eda172a929f7a6508e7d16 | ||
hooks: | ||
- id: go-fmt | ||
files: \.go$ | ||
- id: go-lint | ||
files: \.go$ | ||
exclude: (.*\/client\/.*\.go|.*\generated\.deepcopy\.go)$ | ||
- repo: https://github.com/PaddlePaddle/mirrors-yapf.git | ||
sha: 0d79c0c469bab64f7229c9aca2b1186ef47f0e37 | ||
hooks: | ||
- id: yapf | ||
files: (.*\.(py|bzl)|BUILD|.*\.BUILD|WORKSPACE)$ | ||
- repo: local | ||
hooks: | ||
- id: copyright_checker | ||
name: copyright_checker | ||
entry: python ./.copyright.hook | ||
language: system | ||
files: \.(c|cc|cxx|cpp|cu|h|hpp|hxx|proto|py)$ | ||
- repo: git://github.com/dnephin/pre-commit-golang | ||
sha: HEAD | ||
hooks: | ||
- id: go-fmt | ||
- id: go-vet | ||
- id: go-lint | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
FROM golang:1.8 | ||
RUN go get github.com/Masterminds/glide | ||
RUN apt-get update && apt-get install -y git | ||
WORKDIR $GOPATH/src/github.com/paddlepaddle | ||
RUN git clone https://github.com/paddlepaddle/edl.git | ||
WORKDIR $GOPATH/src/github.com/paddlepaddle/edl | ||
RUN glide install --strip-vendor | ||
RUN go build -o /usr/local/bin/edl github.com/paddlepaddle/edl/cmd/edl | ||
CMD ["edl"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,61 +1,34 @@ | ||
# PaddlePaddle Cloud | ||
# PaddlePaddle EDL: Elastic Deep Learning | ||
|
||
[![Build Status](https://travis-ci.org/PaddlePaddle/cloud.svg?branch=develop)](https://travis-ci.org/PaddlePaddle/cloud) | ||
While many hardware and software manufacturers are working on | ||
improving the running time of deep learning jobs, EDL optimizes | ||
|
||
PaddlePaddle Cloud is a combination of PaddlePaddle and Kubernetes. It | ||
supports fault-recoverable and fault-tolerant large-scaled distributed | ||
deep learning. We can deploy it on public cloud and on-premise | ||
clusters. | ||
1. the global utilization of the cluster, and | ||
1. the waiting time of job submitters. | ||
|
||
PaddlePaddle Cloud includes the following components: | ||
For more about the project EDL, please refer to this [invited blog | ||
post](http://blog.kubernetes.io/2017/12/paddle-paddle-fluid-elastic-learning.html) | ||
on the Kubernetes official blog. | ||
|
||
- paddlectl: A command-line tool that talks to paddlecloud and | ||
paddle-fs. | ||
- paddlecloud: An HTTP server that exposes Kubernetes as a Web | ||
service. | ||
- paddle-fs: An HTTP server that exposes the CephFS distributed | ||
filesystem as a Web service. | ||
- EDL (elastic deep learning): A Kubernetes controller that supports | ||
elastic scheduling of deep learning jobs and other jobs. | ||
- Fault-tolerant distributed deep learning: This part is in | ||
the [Paddle](https://github.com/PaddlePaddle/paddle) repo. | ||
EDL includes two parts: | ||
|
||
## Tutorials | ||
1. a Kubernetes controller for the elastic scheduling of distributed | ||
deep learning jobs, and | ||
|
||
- [快速开始](./doc/tutorial_cn.md) | ||
- [中文手册](./doc/usage_cn.md) | ||
1. making PaddlePaddle a fault-tolerable deep learning framework. | ||
This directory contains the Kubernetes controller. For more | ||
information about fault-tolerance, please refer to the | ||
[design](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/design/cluster_train). | ||
|
||
We deployed EDL on a real Kubernetes cluster, dlnel.com, opened for | ||
graduate students of Tsinghua University. The performance test report | ||
of EDL on this cluster is | ||
[here](https://github.com/PaddlePaddle/cloud/blob/develop/doc/edl/experiment/README.md). | ||
|
||
## How To | ||
|
||
- [Build PaddlePaddle Cloud](./doc/howto/build.md) | ||
- [Deploy PaddlePaddle Cloud](./doc/howto/deploy.md) | ||
- [Elastic Deep Learning using EDL](./doc/edl/example/autoscale.md) | ||
- [PaddlePaddle Cloud on Minikube](./doc/howto/run_on_minikube.md) | ||
## Build | ||
|
||
## Directory structure | ||
|
||
``` | ||
. | ||
├── demo: distributed version of https://github.com/PaddlePaddle/book programs | ||
├── doc: documents | ||
├── docker: scripts to build Docker image to run PaddlePaddle distributed | ||
├── go | ||
│ ├── cmd | ||
│ │ ├── edl: entry of EDL controller binary | ||
│ │ ├── paddlecloud: the command line client of PaddlePaddle Cloud (will be deprecated) | ||
│ │ ├── paddlectl: the command line client of PaddlePaddle Cloud | ||
│ │ └── pfsserver: entry of PaddleFS binary | ||
│ ├── edl: EDL implementation | ||
│ ├── filemanager: PaddleFS implementation | ||
│ ├── paddlecloud: command line client implement (will be deprecated) | ||
│ ├── paddlectl: command line client implement | ||
│ ├── scripts: scripts for Go code generation | ||
├── k8s: YAML files to create different components of PaddlePaddle Cloud | ||
│ ├── edl: TPR definition and EDL controller for TraningJob resource | ||
│ │ ├── autoscale_job: A sample TrainingJob that can scale | ||
│ │ └── autoscale_load: A sample cluster job demonstrating a common workload | ||
│ ├── minikube: YAML files to deploy on local mini-kube environment | ||
│ └── raw_job: A demo job demonstrates how to run PaddlePaddle jobs in cluster | ||
└── python: PaddlePaddle Cloud REST API server | ||
```bash | ||
glide install --strip-vendor | ||
go build -o path/to/output github.com/paddlepaddle/edl/cmd/edl | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
May don't change the copyright owner
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done, thank you!