getting_started_notebooks/intro_tutorials/08_Introduction_to_Dask_XGBoost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"introduction\"></a>\n",
    "## Introduction to Dask XGBoost\n",
    "#### By Paul Hendricks\n",
    "-------\n",
    "\n",
    "In this notebook, we will show how to work with Dask XGBoost in RAPIDS.\n",
    "\n",
    "**Table of Contents**\n",
    "\n",
    "* [Introduction to Dask XGBoost](#introduction)\n",
    "* [Setup](#setup)\n",
    "* [Load Libraries](#libraries)\n",
    "* [Create a Cluster and Client](#cluster)\n",
    "* [Generate Data](#generate)\n",
    "  * [Load Data](#load)\n",
    "  * [Simulate Data](#simulate)\n",
    "  * [Split Data](#split)\n",
    "  * [Check Dimensions](#check)\n",
    "* [Distribute Data using Dask cuDF](#distribute)\n",
    "* [Set Parameters](#parameters)\n",
    "* [Train Model](#train)\n",
    "* [Generate Predictions](#predict)\n",
    "* [Evaluate Model](#evaluate)\n",
    "* [Conclusion](#conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## Setup\n",
    "\n",
    "This notebook was tested using the following Docker containers:\n",
    "\n",
    "* `rapidsai/rapidsai-nightly:0.8-cuda10.0-devel-ubuntu18.04-gcc7-py3.7` from [DockerHub - rapidsai/rapidsai-nightly](https://hub.docker.com/r/rapidsai/rapidsai-nightly)\n",
    "\n",
    "This notebook was run on the NVIDIA Tesla V100 GPU. Please be aware that your system may be different and you may need to modify the code or install packages to run the below examples. \n",
    "\n",
    "If you think you have found a bug or an error, please file an issue here: https://github.com/rapidsai/notebooks/issues\n",
    "\n",
    "Before we begin, let's check out our hardware setup by running the `nvidia-smi` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:38.237293Z",
     "start_time": "2018-11-06T21:03:37.388285Z"
    }
   },
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's see what CUDA version we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:39.490984Z",
     "start_time": "2018-11-06T21:03:39.134608Z"
    }
   },
   "outputs": [],
   "source": [
    "!nvcc --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"libraries\"></a>\n",
    "## Load Libraries\n",
    "\n",
    "Let's load some of the libraries within the RAPIDs ecosystem and see which versions we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:41.067879Z",
     "start_time": "2018-11-06T21:03:40.256654Z"
    }
   },
   "outputs": [],
   "source": [
    "import cudf; print('cuDF Version:', cudf.__version__)\n",
    "import dask; print('Dask Version:', dask.__version__)\n",
    "import dask_cudf; print('Dask cuDF Version:', dask_cudf.__version__)\n",
    "import dask_xgboost; print('Dask XGBoost Version:', dask_xgboost.__version__)\n",
    "import numpy as np; print('numpy Version:', np.__version__)\n",
    "import pandas as pd; print('pandas Version:', pd.__version__)\n",
    "import sklearn; print('Scikit-Learn Version:', sklearn.__version__)\n",
    "# import xgboost as xgb; print('XGBoost Version:', xgb.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cluster\"></a>\n",
    "## Create a Cluster and Client\n",
    "\n",
    "Let's start by creating a local cluster of workers and a client to interact with that cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask.distributed import Client\n",
    "from dask_cuda import LocalCUDACluster\n",
    "\n",
    "\n",
    "# create a local CUDA cluster\n",
    "cluster = LocalCUDACluster()\n",
    "client = Client(cluster)\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"generate\"></a>\n",
    "## Generate Data\n",
    "\n",
    "<a id=\"load\"></a>\n",
    "### Load Data\n",
    "\n",
    "We can load the data using `pandas.read_csv`. We've provided a helper function `load_data` that will load data from a CSV file (and will only read the first 1 billion rows if that file is unreasonably big)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper function for loading data\n",
    "def load_data(filename, n_rows):\n",
    "    if n_rows >= 1e9:\n",
    "        df = pd.read_csv(filename)\n",
    "    else:\n",
    "        df = pd.read_csv(filename, nrows=n_rows)\n",
    "    return df.values.astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"simulate\"></a>\n",
    "### Simulate Data\n",
    "\n",
    "Alternatively, we can simulate data for our train and validation datasets. The features will be tabular with `n_rows` and `n_columns` in the training dataset, where each value is either of type `np.float32`. We can simulate data for both classification and regression using the `make_classification` or `make_regression` functions from the Scikit-Learn package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification, make_regression\n",
    "\n",
    "\n",
    "# helper function for simulating data\n",
    "def simulate_data(m, n, k=2, random_state=None, classification=True):\n",
    "    if classification:\n",
    "        features, labels = make_classification(n_samples=m, n_features=n, \n",
    "                                               n_informative=int(n/5), n_classes=k, \n",
    "                                               random_state=random_state)\n",
    "    else:\n",
    "        features, labels = make_regression(n_samples=m, n_features=n, \n",
    "                                           n_informative=int(n/5), n_targets=1, \n",
    "                                           random_state=random_state)\n",
    "    return np.c_[labels, features].astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# settings\n",
    "simulate = True\n",
    "classification = True  # change this to false to use regression\n",
    "n_rows = int(1e6)  # we'll use 1 millions rows\n",
    "n_columns = int(100)\n",
    "n_categories = 2\n",
    "random_state = np.random.RandomState(43210)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "if simulate:\n",
    "    dataset = simulate_data(n_rows, n_columns, n_categories, \n",
    "                            random_state=random_state, \n",
    "                            classification=classification)\n",
    "else:\n",
    "    dataset = load_data('/tmp', n_rows)\n",
    "print(dataset.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"split\"></a>\n",
    "### Split Data\n",
    "\n",
    "We'll split our dataset into a 80% training dataset and a 20% validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# identify shape and indices\n",
    "n_rows, n_columns = dataset.shape\n",
    "train_size = 0.80\n",
    "train_index = int(n_rows * train_size)\n",
    "\n",
    "# split X, y\n",
    "X, y = dataset[:, 1:], dataset[:, 0]\n",
    "del dataset\n",
    "\n",
    "# split train data\n",
    "X_train, y_train = X[:train_index, :], y[:train_index]\n",
    "\n",
    "# split validation data\n",
    "X_validation, y_validation = X[train_index:, :], y[train_index:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"check\"></a>\n",
    "### Check Dimensions\n",
    "\n",
    "We can check the dimensions and proportions of our training and validation dataets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check dimensions\n",
    "print('X_train: ', X_train.shape, X_train.dtype, 'y_train: ', y_train.shape, y_train.dtype)\n",
    "print('X_validation', X_validation.shape, X_validation.dtype, 'y_validation: ', y_validation.shape, y_validation.dtype)\n",
    "\n",
    "# check the proportions\n",
    "total = X_train.shape[0] + X_validation.shape[0]\n",
    "print('X_train proportion:', X_train.shape[0] / total)\n",
    "print('X_validation proportion:', X_validation.shape[0] / total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"distribute\"></a>\n",
    "### Distribute Data using Dask cuDF\n",
    "\n",
    "Next, let's distribute our data across multiple GPUs using Dask cuDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create Pandas DataFrames for X_train and X_validation\n",
    "n_columns = X_train.shape[1]\n",
    "X_train_pdf = pd.DataFrame(X_train)\n",
    "X_train_pdf.columns = ['feature_' + str(i) for i in range(n_columns)]\n",
    "X_validation_pdf = pd.DataFrame(X_validation)\n",
    "X_validation_pdf.columns = ['feature_' + str(i) for i in range(n_columns)]\n",
    "\n",
    "# create Pandas DataFrames for y_train and y_validation\n",
    "y_train_pdf = pd.DataFrame(y_train)\n",
    "y_train_pdf.columns = ['y']\n",
    "y_validation_pdf = pd.DataFrame(y_validation)\n",
    "y_validation_pdf.columns = ['y']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dask settings\n",
    "npartitions = 8\n",
    "\n",
    "# create Dask DataFrames for X_train and X_validation\n",
    "X_train_dask_pdf = dask.dataframe.from_pandas(X_train_pdf, npartitions=npartitions)\n",
    "X_validation_dask_pdf = dask.dataframe.from_pandas(X_validation_pdf, npartitions=npartitions)\n",
    "\n",
    "# create Dask cuDF DataFrames for X_train and X_validation\n",
    "X_train_dask_cudf = dask_cudf.from_dask_dataframe(X_train_dask_pdf)\n",
    "X_validation_dask_cudf = dask_cudf.from_dask_dataframe(X_validation_dask_pdf)\n",
    "\n",
    "# create Dask DataFrames for y_train and y_validation\n",
    "y_train_dask_pdf = dask.dataframe.from_pandas(y_train_pdf, npartitions=npartitions)\n",
    "y_validation_dask_pdf = dask.dataframe.from_pandas(y_validation_pdf, npartitions=npartitions)\n",
    "\n",
    "# create Dask cuDF DataFrames for y_train and y_validation\n",
    "y_train_dask_cudf = dask_cudf.from_dask_dataframe(y_train_dask_pdf)\n",
    "y_validation_dask_cudf = dask_cudf.from_dask_dataframe(y_validation_dask_pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional: persist training and validation data into memory\n",
    "X_train_dask_cudf = X_train_dask_cudf.persist()\n",
    "X_validation_dask_cudf = X_validation_dask_cudf.persist()\n",
    "y_train_dask_cudf = y_train_dask_cudf.persist()\n",
    "y_validation_dask_cudf = y_validation_dask_cudf.persist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"parameters\"></a>\n",
    "## Set Parameters\n",
    "\n",
    "There are a number of parameters that can be set before XGBoost can be run. \n",
    "\n",
    "* General parameters relate to which booster we are using to do boosting, commonly tree or linear model\n",
    "* Booster parameters depend on which booster you have chosen\n",
    "* Learning task parameters decide on the learning scenario. For example, regression tasks may use different parameters with ranking tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:03:57.443698Z",
     "start_time": "2018-11-06T21:03:57.438288Z"
    }
   },
   "outputs": [],
   "source": [
    "# instantiate params\n",
    "params = {}\n",
    "\n",
    "# general params\n",
    "general_params = {'silent': 1}\n",
    "params.update(general_params)\n",
    "\n",
    "# booster params\n",
    "n_gpus = 1  \n",
    "booster_params = {}\n",
    "booster_params['max_depth'] = 8\n",
    "booster_params['grow_policy'] = 'lossguide'\n",
    "booster_params['max_leaves'] = 2**8\n",
    "booster_params['tree_method'] = 'gpu_hist'\n",
    "booster_params['n_gpus'] = 1  # keep this at 1, even if using more than 1 GPU - Dask XGBoost uses 1 GPU per worker\n",
    "params.update(booster_params)\n",
    "\n",
    "# learning task params\n",
    "learning_task_params = {}\n",
    "if classification:\n",
    "    learning_task_params['eval_metric'] = 'auc'\n",
    "    learning_task_params['objective'] = 'binary:logistic'\n",
    "else:\n",
    "    learning_task_params['eval_metric'] = 'rmse'\n",
    "    learning_task_params['objective'] = 'reg:squarederror'\n",
    "params.update(learning_task_params)\n",
    "print(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"train\"></a>\n",
    "## Train Model\n",
    "\n",
    "Now it's time to train our model! We can use the `dask_xgboost.train` function and pass in the parameters, training dataset, the number of boosting iterations, and the list of items to be evaluated during training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model training settings\n",
    "num_round = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-11-06T21:04:50.201308Z",
     "start_time": "2018-11-06T21:04:00.363740Z"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "\n",
    "bst = dask_xgboost.train(client, params, X_train_dask_cudf, y_train_dask_cudf, num_boost_round=num_round)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"predict\"></a>\n",
    "## Generate Predictions\n",
    "\n",
    "We can generated predictions using the `dask_xgboost.predict` method and then using `dask.dataframe.multi.concat` to concatenate the multiple resulting dataframes together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_predictions = dask_xgboost.predict(client, bst, X_validation_dask_cudf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_predictions = dask.dataframe.multi.concat([y_predictions], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"evaluate\"></a>\n",
    "## Evaluate Model\n",
    "\n",
    "Lastly, we can evaluate our model (depending on classification or regression) and calculate accuracy or rmse, respectively. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "\n",
    "if classification:\n",
    "    thresholded_predictions = (y_predictions[0] > 0.5).compute().to_array() * 1.0\n",
    "    accuracy = accuracy_score(y_validation, thresholded_predictions)\n",
    "    print('Accuracy:', accuracy)\n",
    "else:\n",
    "    test['squared_error'] = (y_predictions[0] - y_validation_dask_cudf['y'])**2\n",
    "    rmse = np.sqrt(test.squared_error.mean().compute())\n",
    "    print('Root Mean Squared Error:', rmse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"conclusion\"></a>\n",
    "## Conclusion\n",
    "\n",
    "In this notebook, we showed how to work with Dask XGBoost in RAPIDS.\n",
    "\n",
    "To learn more about RAPIDS, be sure to check out: \n",
    "\n",
    "* [Open Source Website](http://rapids.ai)\n",
    "* [GitHub](https://github.com/rapidsai/)\n",
    "* [Press Release](https://nvidianews.nvidia.com/news/nvidia-introduces-rapids-open-source-gpu-acceleration-platform-for-large-scale-data-analytics-and-machine-learning)\n",
    "* [NVIDIA Blog](https://blogs.nvidia.com/blog/2018/10/10/rapids-data-science-open-source-community/)\n",
    "* [Developer Blog](https://devblogs.nvidia.com/gpu-accelerated-analytics-rapids/)\n",
    "* [NVIDIA Data Science Webpage](https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}