The directory named "auto_benchmark" has four modules ,they are:
build
: this module is used to build whl package from git code and install the package in a specified image to build a new image , and push this new image to the remote docker hub. It has been put in the directory named "acc_auc_benchmark".
acc_auc_benchmark
:this module is used to benchmark the ACC and AUC of specified deepRec package, which should has been installed in a docker image using the build module already.
gstep_benchmark
: this module is used to benchmark the Global steps/ second of specified deepRec package in a standalone environment, which should has ben installed in a docker image using the build module already.
-
distrib_benchmark
:this module is used to benchmark the Global steps/seconds of specified deepRec package in a distributed environment ,the deepRec package should has ben installed in a docker image using the build module already. -
unit_test
: this module is used to do unit test base on specified branch_name and commit_id
First of all you should clone the code to the host which you want to run you code at.
git clone https://github.com/GosTraight2020/DeepRec_CI.git
- modify the items you need in ./auto_benchmark/acc_auc_benchmark/config.properties, such as branch name, commit id ,ase image et al.
- cd to the directory : ./auto_benchmark/acc_auc_benchmark/build/
cd ./DeepRec_CI/acc_auc_benchmark/build/
- run the script
image_build.sh
with an optional parameter "- l", which tells the program whether you want to tag the image as the latest after the image has been built.
bash image_build.sh [-l]
- cd to the directory "./DeepRec_CI/acc_auc_benchmark/"
cd ./DeepRec_CI/acc_auc_benchmark/
- modify the items you need in ./config.properties, change the value of the key "deeprec_test_image" and "tf_test_image" to be the image you want to use to benchmark.
- modify the command rows that start with "CMD", they are commands which will be implemented in the image.
- run the " ./acc_benchmark.sh"
bash acc_benchmark.sh
- check the log files in the directory "./DeepRec_CI/acc_auc_benchmark/benchmark_result/log/$currentTime"
- cd to the directory "./DeepRec_CI/gstep_benchmark/"
cd ./DeepRec_CI/gstep_benchmark/
- modify the items you need in ./config.properties, change the value of the key "deeprec_test_image" and "tf_test_image" to be the image you want to use to benchmark.
- modify the command rows that start with "CMD", they are commands which will be implemented in the image.
- run the script " ./acc_benchmark.sh"
bash gstep_benchmark.sh
- check the log files in the directory "./DeepRec_CI/gstep_benchmark/benchmark_result/logs/$currentTime"
How to benchmark Global steps/s using an image with installed deepRec environment in distributed env
- cd to the directory "./DeepRec_CI/distrib_benchmark/"
cd ./DeepRec_CI/acc_auc_benchmark/
- you can modify the yaml files which are used to create an k8s tasks by only modifying the template files in directory :".\templates" , and then implement the python script "gen_template.py"
python gen_template.py
- run the script " ./distributed_test.sh"
bash gstep_benchmark.sh
- check the log files in the directory "./DeepRec_CI/distrib_benchmark/logs/$currentTime"
- cd to the directory "./DeepRec/unit_test"
cd DeepRec_CI/unit_test
- modify the config.properties files to specify the parameters you want .including the branch_name and commit_id
- run the script "unit_test.sh"
bash unit_test.sh
- check the ut_res.log and ut_res.xls files in the directory "DeepRec_CI/unit_test/about_ut/log/$currentTime"