aws benchmarking tool #9638
aws benchmarking tool #9638
Conversation
| @@ -0,0 +1 @@ | |||
| nvidia-docker run -i -e "TRAINING_ROLE=PSERVER" -e "PSERVER_HOSTS={PSERVER_HOSTS}" {DOCKER_IMAGE} | |||
There was a problem hiding this comment.
May we need a argument -p to port the container port to host.
|
@Yancey1989 this is a great idea to run this tool in docker, will do, thanks |
There was a problem hiding this comment.
It would be nice if we can achieve:
-
the user don't have to change anything in the
train.py, all they have to do is to provide thetrain.pyand specify the command line arguments for pserver and trainer. -
terminate ec2 instance automatically when the training process finishes (either due to completion, or due to crash).
-
save all output (stdout and stderr) from the process, able to show the output in realtime.
| parser.add_argument( | ||
| '--pserver_instance_type', | ||
| type=str, | ||
| default="p2.8xlarge", |
There was a problem hiding this comment.
We probably don't need GPU instance for pserver.
There was a problem hiding this comment.
agree, will update default pserver instance type
| '--pserver_count', type=int, default=1, help="Pserver count") | ||
|
|
||
| parser.add_argument( | ||
| '--action', type=str, default="serve", help="create|cleanup|status") |
There was a problem hiding this comment.
serve is not in the set "create|cleanup|status"?
There was a problem hiding this comment.
thanks for catching this
|
@helinwang thanks for the great ideas, will update |
|
Going to make some final tweaks and a Readme file tomorrow. |
| ``` | ||
|
|
||
| ***Please Note*** | ||
| Training nodes will run your `ENTRYPOINT` script with the following environment variables: |
There was a problem hiding this comment.
Does this works with our current benchmark scripts?
The vgg16 script takes env variable: SERVER_ENDPOINT PSERVERS TRAINERS TRAINING_ROLE
example usages:
pserver:
SERVER_ENDPOINT=172.19.61.250:8000 PSERVERS=172.19.61.250:8000 TRAINERS=1 TRAINING_ROLE=PSERVER CUDA_VISIBLE_DEVICES=2 LD_LIBRARY_PATH=`pwd`:/usr/local/cuda-8.0/lib64:/usr/local/lib/ python vgg16_fluid.py --local false --device GPU --data_set flowers --batch_size 4
trainer:
SERVER_ENDPOINT=172.19.61.250:8000 PSERVERS=172.19.61.250:8000 TRAINERS=1 TRAINING_ROLE=TRAINER CUDA_VISIBLE_DEVICES=1 LD_LIBRARY_PATH=`pwd`:/usr/local/cuda-8.0/lib64:/usr/local/lib/ python vgg16_fluid.py --local false --device GPU --data_set flowers --batch_size 4
There was a problem hiding this comment.
got it, will update the env vars
tools/aws_benchmarking/README.md
Outdated
| To access the master log: | ||
|
|
||
| ```bash | ||
| docker run -i -v $HOME/.aws:/root/.aws -v <full path to your pem file>:/<key pare name>.pem \ |
There was a problem hiding this comment.
Curious why accessing master log requires *.pem?
There was a problem hiding this comment.
ah, good catch, we don't need pem to access log, will update.
tools/aws_benchmarking/README.md
Outdated
| To retrieve training logs | ||
| TBD | ||
|
|
||
| ### Tech details |
There was a problem hiding this comment.
There is one special character here that needs to be deleted (rendered as �Tech details).
|
|
||
| - `TASK_NAME`: unique name to identify this training process. | ||
| - `TRAINING_ROLE`: current node's role in this training process, either "PSERVER" or "TRAINER" | ||
| - `PSERVER_HOSTS`: comma separated value of pserver end points, I.E. "192.168.1.2:5436,192.168.1.3:5436" |
There was a problem hiding this comment.
Do we need PSERVER_HOSTS? It's not in @typhoonzero 's script or transpiler.py, could we remove it, otherwise it causes confusion like why there are "PSERVER_HOSTS" and "PSERVERS" with the exact same meaning.
There was a problem hiding this comment.
the reason leaving these duplicated env vars is to be compatible with other existing tests, they may require different env vars for the same purpose.
Purpose
This is an automation tool for deploying paddlepaddle benchmark testing to aws.
Features
for more info, please refer to the README.md in this PR.