3. Training Models on Datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Models on Datasets\n",
    "\n",
    "So far, we've seen ways that we can load, explore, and visualize our data. These are the essential steps towards an exciting part of the data science process: modeling! We have to have an understanding of our data before getting to this stage, so now let's dive right in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Using `scikit-learn`\n",
    "\n",
    "We will be using the `scikit-learn` package in order to perform _linear regression_ (more on what that is later). `scikit-learn` is awesome because it has a lot of common machine learning algorithms already built in and it streamlines the process of fitting models and evaluating them on new data. Let's load it up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "\n",
    "# to keep our notebook clean due to the use of some old features\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While it seems pretty simple, we are well on our way to modeling our data: `scikit-learn` is an excellent library full of features that makes modeling data simple and interactive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Linear Regression\n",
    "\n",
    "Let's just briefly go over what kind of model we're fitting. (We'll discuss what other kinds of modeling people usually do in the subsequent sections.) We're trying to predict a real-valued quantity, which gives the regression part of the name, and our model assumes that the output is a linear combination of the inputs, which gives the linear part of the name. For a little bit more math, we have a dataset of $N$ points, and each data point is given by $\\left(x^{(i)}_1, x^{(i)}_2, \\dots x^{(i)}_d, y^{(i)}\\right)$; in other words, each data point has $d$ features or inputs and 1 output. As we said earlier, $y^{(i)}$ is some number we're interested in predicting, so $y^{(i)} \\in \\mathbb{R}.$ The model structure then dictates that we will approximate $y^{(i)}$ in the following way: $$y^{(i)} \\approx \\beta_0 + \\beta_1 x^{(i)}_1 + \\beta_2 x^{(i)}_2 + \\cdots + \\beta_d x^{(i)}_d.$$ Cool, that's the most math we'll have to do today!\n",
    "\n",
    "Let's go ahead and produce our data: we'll keep it simple and make a problem with 1 predictor ($d = 1$) so that we can visualize our results. How will we produce our data? We'll essentially generate data assuming the model is true and add some noise; then, we'll recover the estimates of both $\\beta_0$ and $\\beta_1$, and our model should be pretty close. \n",
    "\n",
    "We'll be using `numpy` to produce the data, but don't worry about following the entire process exactly; you usually won't be using simulated values when making models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generating noisy test data using numpy for demonstration\n",
    "import numpy as np\n",
    "np.random.seed(0)\n",
    "\n",
    "num_data_points = 100\n",
    "xs = np.random.uniform(0, 10, num_data_points)           \n",
    "ys = 3 * xs + 7                                          \n",
    "noisy_ys = ys + np.random.normal(0, 5, num_data_points) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using our data visualization skills, let's see what these data look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# creating a scatter plot\n",
    "plt.scatter(xs, noisy_ys)\n",
    "plt.title(\"Data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, looks like a linear model will do decently well! Let's finish off the data preparation by converting everything into a `pandas` data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# creating a new data frame from our generated data\n",
    "data = pd.DataFrame({'input' : xs, 'output' : noisy_ys})\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Fitting Linear Regression Model with `scikit-learn`\n",
    "\n",
    "We're ready to fit the linear model now! Let's do that using `scikit-learn`. \n",
    "\n",
    "One of the reasons this package is so easy to use is that there is a standard process for how to fit models:\n",
    "\n",
    "    (1) Instantiate model type\n",
    "    (2) Fit the model using your data\n",
    "    (3) Evaluate it on new data\n",
    "    (4) ???\n",
    "    (5) Profit\n",
    "    \n",
    "Well, maybe not steps 4 and 5 necessarily, but definitely the first three! Let's follow these steps for linear regression.\n",
    "\n",
    "Here's step 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "linear_regression_model = LinearRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's step 2:\n",
    "\n",
    "(It's important to note that we're not including the `output` variable as part of the data; otherwise, we could always use that and have perfect prediction accuracy! Of course, when we evaluate future data, we won't have the same luxury of having the right answer ahead of time..)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression_model.fit(data.drop('output', axis = 1), data.output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at what `scikit-learn` did:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Estimated Slope: %.3f (True Slope: 3)\" % linear_regression_model.coef_)\n",
    "print(\"Estimated Intercept: %.3f (True Intercept: 7)\" % linear_regression_model.intercept_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hey, not bad for our first stab at some modeling! Let's visualize our fit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predicting values using our model\n",
    "evaluation_xs = np.linspace(0, 10)\n",
    "evaluation_predicted_ys = linear_regression_model.predict(pd.DataFrame({'input' : evaluation_xs}))\n",
    "\n",
    "# making a scatter plot\n",
    "plt.scatter(xs, noisy_ys)\n",
    "plt.plot(evaluation_xs, evaluation_predicted_ys, color = \"red\")\n",
    "plt.title(\"Visualizing Linear Model Fit\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(In order to visualize the model, we picked some points from $x = 0$ to $x = 50$ using `linspace`, evaluated the predicted output values at those points using the `predict` function, and then plotted the result.)\n",
    "\n",
    "Looks like we did pretty well!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Training Data vs. Testing Data\n",
    "\n",
    "There's something we slightly overlooked in our steps so far; we only have evaluated our model on the same set of points that we used to train our model. That's a little like our professors giving us sample questions ahead of time and then asking the exact same questions on the exam: while it may improve our grades, it won't really evaluate what we know! In the same fashion, we don't want to evaluate our model on just the data we have already seen; therefore, we usually split our data into a _testing_ and _training_ set. We will use the training set to fit our model and use the testing set to fairly evaluate our model performance because the data is \"new\" since the model didn't see it during the model fitting process. `scikit-learn` makes this super easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "\n",
    "training_data, testing_data = train_test_split(data, test_size = 0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool, now we can train on only the training set like we're supposed to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fixed_linear_regression_model = LinearRegression()\n",
    "fixed_linear_regression_model.fit(training_data.drop('output', axis = 1), training_data.output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A typical way to evaluate how well we're doing is something called the coefficient of determination or $R^2$, which ranges from 0 to 1 with values closer to 1 being better; we can evaluate this on both the training and test set to see how they compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data_score = fixed_linear_regression_model.score(training_data.drop('output', axis = 1), training_data.output)\n",
    "testing_data_score = fixed_linear_regression_model.score(testing_data.drop('output', axis = 1), testing_data.output)\n",
    "\n",
    "print(\"Coefficient of Determination for Training Data: %.3f\" % training_data_score)\n",
    "print(\"Coefficient of Determination for Test Data: %.3f\" % testing_data_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, our score on the training data is higher than our score on the testing data; this makes sense since we've seen the training data before. The key lesson to take away is that we should always evaluate our models on unseen data so that we don't trick ourselves into thinking we have a performant model when, in reality, we have just *overfit* the training data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}