Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

GBM-Benchmarks

GBM benchmark suite for the purpose of evaluating the speed of XGBoost GPU on multi-GPU systems with large datasets.

This benchmark is designed to be run on an AWS p3.16xlarge instance with 8 V100 GPUs. It is recommended to use the Deep Learning Base AMI (Ubuntu) and storage of at least 150GB for this task.

Requirements

  • Cuda 9.0
  • Python 3
  • Sklearn, Pandas, numpy
  • Kaggle CLI with a valid API token

Usage

Install XGBoost, LightGBM and Catboost:

sh install_gbm.sh

Run the benchmarks

python3 benchmarks.py

It can be useful to run the benchmarks with a small number of rows/rounds to quickly check everything is working:

python3 benchmarks.py --rows 100 --num_rounds 10

Benchmark parameters:

usage: benchmark.py [-h] [--rows ROWS] [--num_rounds NUM_ROUNDS]
                    [--datasets DATASETS] [--algs ALGS]

optional arguments:
  -h, --help            show this help message and exit
  --rows ROWS           Max rows to benchmark for each dataset. (default:
                        None)
  --num_rounds NUM_ROUNDS
                        Boosting rounds. (default: 500)
  --datasets DATASETS   Datasets to run. (default:
                        YearPredictionMSD,Synthetic,Higgs,Cover
                        Type,Bosch,Airline,)
  --algs ALGS           Boosting algorithms to run. (default: xgb-cpu-
                        hist,xgb-gpu-hist,lightgbm-cpu,lightgbm-gpu,cat-
                        cpu,cat-gpu)

Datasets

Datasets are loaded using ml_dataset_loader. Datasets are automatically downloaded and cached over subsequent runs. Allow time for these downloads on the first run.

Example results

Run on 7 June 2018

"('YearPredictionMSD' 'Time(s)')" "('YearPredictionMSD' 'RMSE')" "('Synthetic' 'Time(s)')" "('Synthetic' 'RMSE')" "('Higgs' 'Time(s)')" "('Higgs' 'Accuracy')" "('Cover Type' 'Time(s)')" "('Cover Type' 'Accuracy')" "('Bosch' 'Time(s)')" "('Bosch' 'Accuracy')" "('Airline' 'Time(s)')" "('Airline' 'Accuracy')"
xgb-cpu-hist 397.27372694015503 8.879391001888838 565.2947809696198 13.610471042735508 470.09188079833984 0.7474345454545455 464.05221605300903 0.891982134712529 752.5890619754791 0.994454065469905 1948.264995098114 0.7494303418939346
xgb-gpu-hist 34.25581908226013 8.879935744972384 38.48715591430664 13.460576927868603 34.07960486412048 0.747475 103.3895480632782 0.8928685145822397 32.12634301185608 0.9944244984160507 144.8635070323944 0.749484266051801
lightgbm-cpu 38.12508988380432 8.877691075962955 421.0538258552551 13.585034611136265 306.9785330295563 0.7473804545454545 83.76876091957092 0.8928340920630277 250.0972819328308 0.9943907074973601 916.0412080287933 0.7504912703697312
lightgbm-gpu 80.04824590682983 8.88175154521266 609.4814240932465 13.585007307447382 529.5377051830292 0.7469995454545455 126.52870297431946 0.8930578384379061 487.14922618865967 0.9944076029567054 614.7447829246521 0.749949160947056
cat-cpu 38.49950695037842 8.994799241732066 436.58789801597595 9.389984249250787 397.02287697792053 0.7406940909090909 288.1107921600342 0.8518626885708631 242.90423798561096 0.9944160506863781 2949.0425968170166 0.7265709745333714
cat-gpu 9.802947044372559 9.036473602545339 35.474628925323486 9.399963630634538 30.145710945129395 0.7406177272727272 N/A N/A N/A N/A 303.35544514656067 0.7277047723183877

Scalability test

We test the scalability of multi-GPU XGBoost by running with between 1-8 GPUs on the airline dataset and timing the results.

python3 scalability.py -h
usage: scalability.py [-h] [--rows ROWS] [--num_rounds NUM_ROUNDS]

optional arguments:
  -h, --help            show this help message and exit
  --rows ROWS           Max rows to benchmark for each dataset. (default:
                        None)
  --num_rounds NUM_ROUNDS
                        Boosting rounds. (default: 500)

About

No description, website, or topics provided.

Resources

License

Releases

No releases published

Packages

No packages published