getting_started_notebooks/intro_tutorials/06_Introduction_to_Supervised_Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"introduction\"></a>\n",
    "## Introduction to Supervised Learning\n",
    "#### By Paul Hendricks\n",
    "-------\n",
    "\n",
    "In this notebook, we will show to do GPU accelerated Supervised Learning in RAPIDS.  We will not cover SGD Regression at this time.\n",
    "\n",
    "**Table of Contents**\n",
    "\n",
    "* [Introduction to Supervised Learning](#introduction)\n",
    "* [Linear Regression](#linear)\n",
    "* [Ridge Regression](#ridge)\n",
    "* [K Nearest Neighbors](#knn)\n",
    "* [Setup](#setup)\n",
    "* [Conclusion](#conclusion)\n",
    "\n",
    "Before going any further, let's make sure we have access to `matplotlib`, a popular Python library for visualizing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import subprocess\n",
    "\n",
    "try:\n",
    "    import matplotlib\n",
    "except ModuleNotFoundError:\n",
    "    os.system('conda install -y matplotlib')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## Setup\n",
    "\n",
    "This notebook was tested using the following Docker containers:\n",
    "\n",
    "* `rapidsai/rapidsai-dev-nightly:0.10-cuda10.0-devel-ubuntu18.04-py3.7` container from [DockerHub](https://hub.docker.com/r/rapidsai/rapidsai-nightly)\n",
    "\n",
    "This notebook was run on the NVIDIA GV100 GPU. Please be aware that your system may be different and you may need to modify the code or install packages to run the below examples. \n",
    "\n",
    "If you think you have found a bug or an error, please file an issue here: https://github.com/rapidsai/notebooks-contrib/issues\n",
    "\n",
    "Before we begin, let's check out our hardware setup by running the `nvidia-smi` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's see what CUDA version we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvcc --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's load some helper functions from `matplotlib` and configure the Jupyter Notebook for visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib.colors import ListedColormap\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"linear\"></a>\n",
    "## Linear Regression\n",
    "\n",
    "After our data has been preprocessed, we often want to build a model so as to understand the relationships between different variables in our data. Scikit-Learn is an incredibly powerful toolkit that allows data scientists to quickly build models from their data. Below we show a simple example of how to create a Linear Regression model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np; print('NumPy Version:', np.__version__)\n",
    "\n",
    "\n",
    "# create the relationship: y = 2.0 * x + 1.0\n",
    "n_rows = 100000  # let's use 100 thousand data points\n",
    "w = 2.0\n",
    "x = np.random.normal(loc=0, scale=1, size=(n_rows,))\n",
    "b = 1.0\n",
    "y = w * x + b\n",
    "\n",
    "# add a bit of noise\n",
    "noise = np.random.normal(loc=0, scale=2, size=(n_rows,))\n",
    "y_noisy = y + noise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now visualize our data using the `matplotlib` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(x, y_noisy, label='empirical data points')\n",
    "plt.plot(x, y, color='black', label='true relationship')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the `LinearRegression` class from Scikit-Learn to instantiate a model and fit it to our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn; print('Scikit-Learn Version:', sklearn.__version__)\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "\n",
    "# instantiate and fit model\n",
    "linear_regression = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "linear_regression.fit(np.expand_dims(x, 1), y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create new data and perform inference\n",
    "inputs = np.linspace(start=-5, stop=5, num=1000)\n",
    "outputs = linear_regression.predict(np.expand_dims(inputs, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now visualize our empirical data points, the true relationship of the data, and the relationship estimated by the model. Looks pretty close!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(x, y_noisy, label='empirical data points')\n",
    "plt.plot(x, y, color='black', label='true relationship')\n",
    "plt.plot(inputs, outputs, color='red', label='predicted relationship (cpu)')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mathematical operations underlying many machine learning algorithms are often matrix multiplications. These types of operations are highly parallelizable and can be greatly accelerated using a GPU. cuML makes it easy to build machine learning models in an accelerated fashion while still using an interface nearly identical to Scikit-Learn. The below shows how to accomplish the same Linear Regression model but on a GPU.\n",
    "\n",
    "First, let's convert our data from a NumPy representation to a cuDF representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cudf; print('cuDF Version:', cudf.__version__)\n",
    "\n",
    "\n",
    "# create a cuDF DataFrame\n",
    "df = cudf.DataFrame({'x': x, 'y': y_noisy})\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll load the GPU accelerated `LinearRegression` class from cuML, instantiate it, and fit it to our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cuml; print('cuML Version:', cuml.__version__)\n",
    "from cuml.linear_model import LinearRegression as LinearRegression_GPU\n",
    "\n",
    "\n",
    "# instantiate and fit model\n",
    "linear_regression_gpu = LinearRegression_GPU()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "linear_regression_gpu.fit(df[['x']], df['y'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this model to predict values for new data points, a step often called \"inference\" or \"scoring\". All model fitting and predicting steps are GPU accelerated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create new data and perform inference\n",
    "new_data_df = cudf.DataFrame({'inputs': inputs})\n",
    "outputs_gpu = linear_regression_gpu.predict(new_data_df[['inputs']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can overlay our predicted relationship using our GPU accelerated Linear Regression model (green line) over our empirical data points (light blue circles), the true relationship (blue line), and the predicted relationship from a model built on the CPU (red line). We see that our GPU accelerated model's estimate of the true relationship (green line) is identical to the CPU based model's estimate of the true relationship (red line)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(x, y_noisy, label='empirical data points')\n",
    "plt.plot(x, y, color='black', label='true relationship')\n",
    "plt.plot(inputs, outputs, color='red', label='predicted relationship (cpu)')\n",
    "plt.plot(inputs, outputs_gpu.to_array(), color='green', label='predicted relationship (gpu)')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ridge\"></a>\n",
    "## Ridge Regression\n",
    "\n",
    "Ridge extends LinearRegression by providing L2 regularization on the coefficients when predicting response y with a linear combination of the predictors in X. It can reduce the variance of the predictors, and improves the conditioning of the problem.\n",
    "\n",
    "Below, we instantiate and fit a Ridge Regression model to our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cuml.linear_model import Ridge as Ridge_GPU\n",
    "\n",
    "\n",
    "# instantiate and fit model\n",
    "ridge_regression_gpu = Ridge_GPU()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "ridge_regression_gpu.fit(df[['x']], df['y'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the `LinearRegression` model we fitted early, we can use the `predict` method to generate predictions for new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "outputs_gpu = ridge_regression_gpu.predict(new_data_df[['inputs']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can visualize our `Ridge` model's estimated relationship and overlay it our the empirical data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(x, y_noisy, label='empirical data points')\n",
    "plt.plot(x, y, color='black', label='true relationship')\n",
    "plt.plot(inputs, outputs, color='red', label='linear regression (cpu)')\n",
    "plt.plot(inputs, outputs_gpu.to_array(), color='green', label='ridge regression (gpu)')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"knn\"></a>\n",
    "## K Nearest Neighbors\n",
    "\n",
    "NearestNeighbors is a unsupervised algorithm where if one wants to find the “closest” datapoint(s) to new unseen data, one can calculate a suitable “distance” between each and every point, and return the top K datapoints which have the smallest distance to it.\n",
    "\n",
    "We'll generate some fake data using the `make_moons` function from the `sklearn.datasets` module. This function generates data points from two equations, each describing a half circle with a unique center. Since each data point is generated by one of these two equations, the cluster each data point belongs to is clear. The ideal classification algorithm will identify two clusters and associate each data point with the equation that generated it. \n",
    "\n",
    "These data points are generated using a non-linear relationship - so using a linear regression approach won't adequately solve problem. Instead, we can use a distance-based algorithm K Nearest Neighbors to classify each data point.\n",
    "\n",
    "First, let's generate out data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "\n",
    "\n",
    "X, y = make_moons(n_samples=int(1e3), noise=0.05, random_state=0)\n",
    "print(X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure = plt.figure()\n",
    "axis = figure.add_subplot(111)\n",
    "axis.scatter(X[y == 0, 0], X[y == 0, 1], \n",
    "             edgecolor='black',\n",
    "             c='lightblue', marker='o', s=40, label='cluster 1')\n",
    "\n",
    "axis.scatter(X[y == 1, 0], X[y == 1, 1], \n",
    "             edgecolor='black',\n",
    "             c='red', marker='s', s=40, label='cluster 2')\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we build a KNN classification model, we first have to convert our data to a cuDF representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_df = cudf.DataFrame()\n",
    "for column in range(X.shape[1]):\n",
    "    X_df['feature_' + str(column)] = np.ascontiguousarray(X[:, column])\n",
    "\n",
    "y_df = cudf.Series(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll instantiate and fit a nearest neighbors model using the `NearestNeighbors` class from cuML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cuml.neighbors import NearestNeighbors\n",
    "\n",
    "\n",
    "knn = NearestNeighbors()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn.fit(X_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once our model has been built and fitted to the data, we can query the model for the `k` nearest neighbors to each data point. The query returns a matrix representating the distances of each data point to its nearest `k` neighbors as well as the indices of those neighbors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_neighbors = 3\n",
    "\n",
    "distances, indices = knn.kneighbors(X_df, k=n_neighbors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can iterate through each of our data points and do a majority vote to determine which class it belongs to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = []\n",
    "\n",
    "for i in range(indices.shape[0]):\n",
    "    row = indices.iloc[i, :]\n",
    "    vote = sum(y_df[j] for j in row) / n_neighbors\n",
    "    predictions.append(1.0 * (vote > 0.5))\n",
    "\n",
    "predictions = np.asarray(predictions).astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can visualize the predictions from our K Nearest Neighbors classifier - we see that despite the non-linearity of the data, the algorithm does an excellent job of classifying the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, (ax1, ax2) = plt.subplots(1, 2, figsize=(8, 3))\n",
    "\n",
    "\n",
    "ax1.scatter(X[y == 0, 0], X[y == 0, 1],\n",
    "            edgecolor='black',\n",
    "            c='lightblue', marker='o', s=40, label='cluster 1')\n",
    "ax1.scatter(X[y == 1, 0], X[y == 1, 1],\n",
    "            edgecolor='black',\n",
    "            c='red', marker='s', s=40, label='cluster 2')\n",
    "ax1.set_title('empirical data points')\n",
    "\n",
    "\n",
    "ax2.scatter(X[predictions == 0, 0], X[predictions == 0, 1], c='lightblue',\n",
    "            edgecolor='black',\n",
    "            marker='o', s=40, label='cluster 1')\n",
    "ax2.scatter(X[predictions == 1, 0], X[predictions == 1, 1], c='red',\n",
    "            edgecolor='black',\n",
    "            marker='s', s=40, label='cluster 2')\n",
    "ax2.set_title('KNN predicted classes')\n",
    "\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"conclusion\"></a>\n",
    "## Conclusion\n",
    "\n",
    "In this notebook, we showed to do GPU accelerated Supervised Learning in RAPIDS. \n",
    "\n",
    "To learn more about RAPIDS, be sure to check out: \n",
    "\n",
    "* [Open Source Website](http://rapids.ai)\n",
    "* [GitHub](https://github.com/rapidsai/)\n",
    "* [Press Release](https://nvidianews.nvidia.com/news/nvidia-introduces-rapids-open-source-gpu-acceleration-platform-for-large-scale-data-analytics-and-machine-learning)\n",
    "* [NVIDIA Blog](https://blogs.nvidia.com/blog/2018/10/10/rapids-data-science-open-source-community/)\n",
    "* [Developer Blog](https://devblogs.nvidia.com/gpu-accelerated-analytics-rapids/)\n",
    "* [NVIDIA Data Science Webpage](https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}