Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove all code besides EDL code; update EDL import path #5

Merged
merged 2 commits into from
Mar 8, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 3 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
*~
vendor/
.glide/
9 changes: 2 additions & 7 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,3 @@
.Python
*.crt
.cache
vendor
*~
*.pyc
*.idea
.vscode
vendor/
.glide/
27 changes: 7 additions & 20 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,20 +1,7 @@
- repo: https://github.com/dnephin/pre-commit-golang
sha: e4693a4c282b4fc878eda172a929f7a6508e7d16
hooks:
- id: go-fmt
files: \.go$
- id: go-lint
files: \.go$
exclude: (.*\/client\/.*\.go|.*\generated\.deepcopy\.go)$
- repo: https://github.com/PaddlePaddle/mirrors-yapf.git
sha: 0d79c0c469bab64f7229c9aca2b1186ef47f0e37
hooks:
- id: yapf
files: (.*\.(py|bzl)|BUILD|.*\.BUILD|WORKSPACE)$
- repo: local
hooks:
- id: copyright_checker
name: copyright_checker
entry: python ./.copyright.hook
language: system
files: \.(c|cc|cxx|cpp|cu|h|hpp|hxx|proto|py)$
- repo: git://github.com/dnephin/pre-commit-golang
sha: HEAD
hooks:
- id: go-fmt
- id: go-vet
- id: go-lint

18 changes: 3 additions & 15 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,13 @@ matrix:
go: 1.8.x
sudo: required
install:
- sudo apt-get install -y mercurial
- go get -u github.com/golang/lint/golint
- curl https://glide.sh/get | bash
- sudo pip install pre-commit
script:
- rm -f .copyright.hook && wget https://raw.githubusercontent.com/PaddlePaddle/Paddle/develop/.copyright.hook
- bash -x .tools/check_style.sh
- ln -s $GOPATH/src/github.com/PaddlePaddle $GOPATH/src/github.com/paddlepaddle
- mkdir ~/.glide && glide mirror set https://gonum.org/v1/plot https://github.com/gonum/plot
- glide install --strip-vendor && go test $(glide novendor)
- language: python
python: 2.7
sudo: required
env:
- DJANGO_SETTINGS_MODULE="paddlecloud.travis_settings"
before_script:
- mysql -e 'create database paddlecloud;'
- mkdir $HOME/.kube && cp ./k8s/config $HOME/.kube/
- pip install -r python/paddlecloud/requirements.txt
- cd python/paddlecloud && python manage.py makemigrations && python manage.py migrate
script:
- python manage.py test
- cd $GOPATH/src/github.com/paddlepaddle/edl
- glide install --strip-vendor
- go test $(glide novendor)
9 changes: 9 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM golang:1.8
RUN go get github.com/Masterminds/glide
RUN apt-get update && apt-get install -y git
WORKDIR $GOPATH/src/github.com/paddlepaddle
RUN git clone https://github.com/paddlepaddle/edl.git
WORKDIR $GOPATH/src/github.com/paddlepaddle/edl
RUN glide install --strip-vendor
RUN go build -o /usr/local/bin/edl github.com/paddlepaddle/edl/cmd/edl
CMD ["edl"]
3 changes: 1 addition & 2 deletions LICENSE
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved

Apache License
Version 2.0, January 2004
Expand Down Expand Up @@ -188,7 +187,7 @@ Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.
Copyright [yyyy] [name of copyright owner]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May don't change the copyright owner

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, thank you!


Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand Down
73 changes: 23 additions & 50 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,61 +1,34 @@
# PaddlePaddle Cloud
# PaddlePaddle EDL: Elastic Deep Learning

[![Build Status](https://travis-ci.org/PaddlePaddle/cloud.svg?branch=develop)](https://travis-ci.org/PaddlePaddle/cloud)
While many hardware and software manufacturers are working on
improving the running time of deep learning jobs, EDL optimizes

PaddlePaddle Cloud is a combination of PaddlePaddle and Kubernetes. It
supports fault-recoverable and fault-tolerant large-scaled distributed
deep learning. We can deploy it on public cloud and on-premise
clusters.
1. the global utilization of the cluster, and
1. the waiting time of job submitters.

PaddlePaddle Cloud includes the following components:
For more about the project EDL, please refer to this [invited blog
post](http://blog.kubernetes.io/2017/12/paddle-paddle-fluid-elastic-learning.html)
on the Kubernetes official blog.

- paddlectl: A command-line tool that talks to paddlecloud and
paddle-fs.
- paddlecloud: An HTTP server that exposes Kubernetes as a Web
service.
- paddle-fs: An HTTP server that exposes the CephFS distributed
filesystem as a Web service.
- EDL (elastic deep learning): A Kubernetes controller that supports
elastic scheduling of deep learning jobs and other jobs.
- Fault-tolerant distributed deep learning: This part is in
the [Paddle](https://github.com/PaddlePaddle/paddle) repo.
EDL includes two parts:

## Tutorials
1. a Kubernetes controller for the elastic scheduling of distributed
deep learning jobs, and

- [快速开始](./doc/tutorial_cn.md)
- [中文手册](./doc/usage_cn.md)
1. making PaddlePaddle a fault-tolerable deep learning framework.
This directory contains the Kubernetes controller. For more
information about fault-tolerance, please refer to the
[design](https://github.com/PaddlePaddle/Paddle/tree/develop/doc/design/cluster_train).

We deployed EDL on a real Kubernetes cluster, dlnel.com, opened for
graduate students of Tsinghua University. The performance test report
of EDL on this cluster is
[here](https://github.com/PaddlePaddle/cloud/blob/develop/doc/edl/experiment/README.md).

## How To

- [Build PaddlePaddle Cloud](./doc/howto/build.md)
- [Deploy PaddlePaddle Cloud](./doc/howto/deploy.md)
- [Elastic Deep Learning using EDL](./doc/edl/example/autoscale.md)
- [PaddlePaddle Cloud on Minikube](./doc/howto/run_on_minikube.md)
## Build

## Directory structure

```
.
├── demo: distributed version of https://github.com/PaddlePaddle/book programs
├── doc: documents
├── docker: scripts to build Docker image to run PaddlePaddle distributed
├── go
│   ├── cmd
│   │   ├── edl: entry of EDL controller binary
│   │   ├── paddlecloud: the command line client of PaddlePaddle Cloud (will be deprecated)
│   │   ├── paddlectl: the command line client of PaddlePaddle Cloud
│   │   └── pfsserver: entry of PaddleFS binary
│   ├── edl: EDL implementation
│   ├── filemanager: PaddleFS implementation
│   ├── paddlecloud: command line client implement (will be deprecated)
│   ├── paddlectl: command line client implement
│   ├── scripts: scripts for Go code generation
├── k8s: YAML files to create different components of PaddlePaddle Cloud
│   ├── edl: TPR definition and EDL controller for TraningJob resource
│   │   ├── autoscale_job: A sample TrainingJob that can scale
│   │   └── autoscale_load: A sample cluster job demonstrating a common workload
│   ├── minikube: YAML files to deploy on local mini-kube environment
│   └── raw_job: A demo job demonstrates how to run PaddlePaddle jobs in cluster
└── python: PaddlePaddle Cloud REST API server
```bash
glide install --strip-vendor
go build -o path/to/output github.com/paddlepaddle/edl/cmd/edl
```
4 changes: 2 additions & 2 deletions go/cmd/edl/edl.go → cmd/edl/edl.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ import (
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"

"github.com/PaddlePaddle/cloud/go/edl"
edlresource "github.com/PaddlePaddle/cloud/go/edl/resource"
"github.com/paddlepaddle/edl/pkg"
edlresource "github.com/paddlepaddle/edl/pkg/resource"
)

func main() {
Expand Down
147 changes: 0 additions & 147 deletions demo/fit_a_line/train.py

This file was deleted.

Loading