labs/lab08/lab.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45f9ebc8",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# Initialize Otter\n",
    "import otter\n",
    "grader = otter.Notebook(\"lab.ipynb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c39b00d3",
   "metadata": {},
   "source": [
    "# Lab 8 – Feature Engineering ⚙️\n",
    "\n",
    "## DSC 80, Winter 2024\n",
    "\n",
    "### Due Date: Monday, March 4, 2024 at 11:59pm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12055b96",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "Welcome to the eighth DSC 80 lab this quarter!\n",
    "\n",
    "Much like in DSC 10, this Jupyter Notebook contains the statements of the problems and provides code and Markdown cells to display your answers to the problems. Unlike DSC 10, the notebook is *only* for displaying a readable version of your final answers. The coding will be done in an accompanying `lab.py` file that is imported into the current notebook, and **you will only submit that `lab.py` file**, not this notebook!\n",
    "\n",
    "Some additional guidelines:\n",
    "- **Unlike in DSC 10, labs will have both public tests and hidden tests.** The bulk of your grade will come from your scores on hidden tests, which you will only see on Gradescope after the assignment deadline.\n",
    "- **Do not change the function names in the `lab.py` file!** The functions in the `lab.py` file are how your assignment is graded, and they are graded by their name. If you changed something you weren't supposed to, you can find the original code in the [course GitHub repository](https://github.com/dsc-courses/dsc80-2024-wi).\n",
    "- Notebooks are nice for testing and experimenting with different implementations before designing your function in your `lab.py` file. You can write code here, but make sure that all of your real work is in the `lab.py` file, since that's all you're submitting.\n",
    "- You are encouraged to write your own additional helper functions to solve the lab, as long as they also end up in `lab.py`.\n",
    "\n",
    "**To ensure that all of the work you want to submit is in `lab.py`, we've included a script named `lab-validation.py` in the lab folder. You shouldn't edit it, but instead, you should call it from the command line (e.g. the Terminal) to test your work.** More details on its usage are given at the bottom of this notebook.\n",
    "\n",
    "**Importing code from `lab.py`**:\n",
    "\n",
    "* Below, we import the `.py` file that's contained in the same directory as this notebook.\n",
    "* We use the `autoreload` notebook extension to make changes to our `lab.py` file immediately available in our notebook. Without this extension, we would need to restart the notebook kernel to see any changes to `lab.py` in the notebook.\n",
    "    - `autoreload` is necessary because, upon import, `lab.py` is compiled to bytecode (in the directory `__pycache__`). Subsequent imports of `lab` merely import the existing compiled python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e51dc377",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2e4d9b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lab import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afd1e6db",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import plotly.express as px\n",
    "import statsmodels.api as sm\n",
    "from pathlib import Path\n",
    "from sklearn.preprocessing import Binarizer, QuantileTransformer, FunctionTransformer\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fd40947",
   "metadata": {},
   "source": [
    "## Part 1: Scaling Transformations 📐\n",
    "\n",
    "A scaling transformation transforms the scale of the data of a particular quantitative column. Mathematically, each data point $d_i$ is replaced with a transformed value $t_i = f(d_i)$, where $f$ is a transformation function. We can transform any column in a dataset, whether it corresponds to a feature or a target.\n",
    "\n",
    "Generally, the goal of a scaling transformation is to change the data from a complicated, non-linear relationship into a **linear** relationship. Linear relationships are very easy to understand and are easily used by models, like linear regression.\n",
    "\n",
    "However, non-linear growth is a commonly seen relationship in data. Sometimes this growth is by a **fixed power** and sometimes it is **exponential**. The scaling transformations that turn these types of growth linear are **root** and **log** transformations respectively. (Generally, it is more difficult to determine which transformation is appropriate for a given dataset, though the [Tukey-Mosteller bulge diagram](https://freakonometrics.hypotheses.org/files/2014/06/Selection_005.png) is useful.)\n",
    "\n",
    "Let's start by looking at some examples of transformations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "572c0716",
   "metadata": {},
   "source": [
    "#### Example 1\n",
    "\n",
    "Run the cell below to generate a scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69fbbb6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# By setting a seed, we guarantee that we will see the same results each time we run this cell.\n",
    "np.random.seed(23)\n",
    "\n",
    "# Generates a random scatter plot\n",
    "x = np.arange(1, 101) + np.random.normal(0, 0.5, 100)\n",
    "y = 2 * ((x + np.random.normal(0, 1, 100)) ** 2) + np.abs(x) * np.random.normal(0, 30, 100)\n",
    "df_1 = pd.DataFrame().assign(x=x, y=y)\n",
    "\n",
    "px.scatter(df_1, x='x', y='y', trendline=\"ols\", trendline_color_override=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c6a0355",
   "metadata": {},
   "source": [
    "It doesn't appear to be the case that `'x'` and `'y'` are linearly associated here, and they aren't – there is a **quadratic** relationship between them. Note that if we were to create a **residual plot** above, there would be a pattern – the residuals for smaller `'x'` would mostly be positive, and the residuals for larger `'x'` would mostly be negative. From [DSC 10](https://inferentialthinking.com/chapters/15/5/Visual_Diagnostics.html), we know that patterns in a residual plot imply that the relationship between the two variables is non-linear.\n",
    "\n",
    "To linearize the relationship, we can take the square root of each `'y'` value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10a93ef2",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_1['root y'] = np.sqrt(df_1['y'])\n",
    "\n",
    "px.scatter(df_1, x='x', y='root y', trendline=\"ols\", trendline_color_override=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "889e0ee6",
   "metadata": {},
   "source": [
    "That looks much better!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f6a77fd",
   "metadata": {},
   "source": [
    "#### Example 2\n",
    "\n",
    "Run the cell below to generate another scatter plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a047b13",
   "metadata": {},
   "outputs": [],
   "source": [
    "# By setting a seed, we guarantee that we will see the same results each time we run this cell\n",
    "np.random.seed(32)\n",
    "\n",
    "# Generates a different random scatter plot\n",
    "x = np.linspace(2, 5, 100)\n",
    "y = 10 * (np.e ** x) + np.abs(x) * np.random.normal(0, 5, 100) + np.random.normal(0, 30, 100)\n",
    "df_2 = pd.DataFrame().assign(x=x, y=y)\n",
    "\n",
    "px.scatter(df_2, x='x', y='y', trendline=\"ols\", trendline_color_override=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e8b8bf9",
   "metadata": {},
   "source": [
    "Again, the relationship between `'x'` and `'y'` is not quite linear. Let's try the square root transformation we tried in Example 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d920625",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2['root y'] = np.sqrt(df_2['y'])\n",
    "\n",
    "px.scatter(df_2, x='x', y='root y', trendline=\"ols\", trendline_color_override=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c699a8d0",
   "metadata": {},
   "source": [
    "Hmm... the relationship certainly looks _more_ linear than before, but still not quite linear. Let's look at the residual plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fbc31b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feel free to use this function directly to help you answer Question 1.\n",
    "def create_residual_plot(df, x, y):\n",
    "    df = df.copy()\n",
    "    from sklearn.linear_model import LinearRegression\n",
    "    model = LinearRegression()\n",
    "    model.fit(df[[x]], df[y])\n",
    "    df['pred'] = model.predict(df[[x]])\n",
    "    df[f'{y} residuals'] = df[y] - model.predict(df[[x]])\n",
    "    return px.scatter(df, x='pred', y=f'{y} residuals', trendline='ols', trendline_color_override='red')\n",
    "\n",
    "create_residual_plot(df_2, 'x', 'root y')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fb359da",
   "metadata": {},
   "source": [
    "There is clearly a pattern in the residual plot. Let's instead try another transformation for the `'y'` values – $\\log$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65308c52",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2['log y'] = np.log(df_2['y'])\n",
    "\n",
    "px.scatter(df_2, x='x', y='log y', trendline=\"ols\", trendline_color_override=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04790cb9",
   "metadata": {},
   "source": [
    "That looks much better! We can verify that the residual plot has no \"patterns\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d068d356",
   "metadata": {},
   "outputs": [],
   "source": [
    "create_residual_plot(df_2, 'x', 'log y')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3e6e139",
   "metadata": {},
   "source": [
    "Note – there is still evidence of **heteroscedasticity**, or \"uneven spread\", in this scatter plot, but the relationship is as close to linear as we'll get."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c80a8623",
   "metadata": {},
   "source": [
    "### Question 1 –  Root vs. Log\n",
    "\n",
    "Now that we've learned how to perform transformations with example datasets, it's your job to apply these ideas to a real dataset. Below, you are given a dataset that describes the [number of home runs in the MLB per year](https://www.mlb.com/glossary/standard-stats/home-run). The relationship between the two variables, `'Year'` and `'Homeruns'`, is not linear.\n",
    "\n",
    "**Specifically, your job is to determine what the appropriate transformation to apply to the `'Home runs'` column is, in order to linearize the relationship.**\n",
    "\n",
    "Complete the implementation of the function `best_transformation`, which returns either 1, 2, 3, or 4, with the value corresponding to one of the following choices:\n",
    "\n",
    "1. Square root transformation.\n",
    "2. Log transformation.\n",
    "3. Both work the same.\n",
    "4. Neither gives a transformation revealing a linear relationship.\n",
    "\n",
    "***Hint***: If you find that both residual plots have some sort of pattern, choose the residual plot in which the vertical spread is constant. There is one clearly correct answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33cce179",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "homeruns_fp = Path('data')/'homeruns.csv'\n",
    "homeruns = pd.read_csv(homeruns_fp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1838108",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80cf53f7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f35bac5e",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d172e36",
   "metadata": {},
   "source": [
    "## Part 2: Diamond Pricing 💎\n",
    "\n",
    "In this next section, you will pretend you are a jewelry appraiser and predict the prices of diamonds given several standard characteristics of diamonds.\n",
    "\n",
    "You will use linear regression to predict prices, while improving the quality of your predictions using **feature engineering**. Since this question is supposed to help you understand feature engineering, **you will be building these features from scratch, instead of using the built in `sklearn` or `pandas` methods**.\n",
    "\n",
    "The `diamonds` dataset is accessible via `seaborn` (with `sns.load_dataset('diamonds')`), but we've skipped that step and loaded it for you below. The DataFrame has 53940 rows and 10 columns:\n",
    "\n",
    "|column|description|unique values or range|\n",
    "|---|---|---|\n",
    "|`'carat'`|weight of the diamond in carats (each carat is 0.2 grams)| 0.2 - 5.01 |\n",
    "|`'cut'`|quality of the cut | Fair, Good, Very Good, Premium, Ideal |\n",
    "|`'color'`|diamond colour | J (worst, near colorless), I, H, G, F, E, D (best, absolute colorless) |\n",
    "|`'clarity'`|a measurement of how clear the diamond is | I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best) |\n",
    "|`'depth'`|total depth percentage, computed as z / mean(x, y) = 2 * z / (x + y) | 43 - 79 |\n",
    "|`'table'`|width of top of diamond relative to widest point | 43 - 95 |\n",
    "|`'price'`|price in US dollars | \\\\$326 - \\\\$18,823 USD |\n",
    "|`'x'`|length in mm | 0 - 10.74 |\n",
    "|`'y'`|width in mm | 0 - 58.9 | \n",
    "|`'z'`|depth in mm | 0 - 31.8 |\n",
    "\n",
    "If you want to learn more about how diamonds are measured, refer to [this page by the American Gem Society](https://www.americangemsociety.org/4cs-of-diamonds/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b85ff994",
   "metadata": {},
   "outputs": [],
   "source": [
    "diamonds = pd.read_csv(Path('data')/'diamonds.csv')\n",
    "diamonds.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a489cb11",
   "metadata": {},
   "source": [
    "### Question 2 – Ordinal Encoding 🔢\n",
    "\n",
    "Every categorical variable in the dataset is an ordinal column, meaning that there is an inherent order that we can use to sort the values in the column. Recall that **ordinal encoding** is a feature transformation that maps the values in an ordinal column to positive integers in a way that preserves the order of the column values. For instance, an ordinal encoding for Freshman, Sophomore, Junior, Senior is 0, 1, 2, 3.\n",
    "\n",
    "Complete the implementation of the function `create_ordinal`, which takes in the `diamonds` DataFrame and returns a DataFrame of ordinal features only with names of the form `'ordinal_<col>'`, where `'<col>'` is the original categorical column name. For instance, the `'ordinal_color'` column should consist of values from 0 to 6, where 0 refers to `'J'` and 6 refers to `'D'`. (In all cases, start counting from 0.)\n",
    "\n",
    "***Notes:*** \n",
    "- Remember, you are creating this function using basic `pandas`. You might want to create a helper function that takes in a single column and an ordering for that column.\n",
    "- Don't include non-ordinal features in the returned DataFrame. That is, if there are only three columns in `diamonds` that are ordinal, `create_ordinal` should return a DataFrame with three columns.\n",
    "- The orderings for each of the ordinal columns are displayed in the data dictionary above (in the `'unique values or range'` column)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e211f857",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8273c3ee",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d900a811",
   "metadata": {},
   "outputs": [],
   "source": [
    "# don't change this cell, but do run it -- it is needed for the tests to work\n",
    "diamonds = pd.read_csv(Path('data')/'diamonds.csv')\n",
    "out_q2 = create_ordinal(diamonds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "240db2b1",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "138eb10f",
   "metadata": {},
   "source": [
    "### Question 3 – Nominal Encoding 📊\n",
    "\n",
    "Even though the categorical variables in the dataset are ordinal, we can still treat them as nominal by forgetting their order. To treat the categorical variables in our dataset as nominal, we might **one-hot encode** them. \n",
    "\n",
    "#### `create_one_hot`\n",
    "\n",
    "Complete the implementation of the function `create_one_hot`, which takes in the `diamonds` DataFrame and returns a DataFrame of one-hot encoded features with names of the form `'one_hot_<col>_<val>'`, where `'<col>'` is the original categorical column name, and `'<val>'` is the value found in the categorical column `'<col>'`. For instance, one of your column names will be `'one_hot_color_J'`.\n",
    "\n",
    "***Notes***:\n",
    "- Only include one-hot-encoded columns in the DataFrame that `create_one_hot` returns.\n",
    "- Create a helper function that creates the one-hot encoding for a single column. **Do not** use `sklearn` or `pd.get_dummies` for this question!\n",
    "- As per usual, write an efficient implementation. You may use a `for`-loop over **columns**, but not over rows. And the order of **columns** does not matter.\n",
    "- In lecture, we will look at cases where we need to drop one one-hot encoded column per categorical variable. **Do not drop** any one-hot encoded columns here!\n",
    "\n",
    "<br>\n",
    "\n",
    "#### `create_proportions`\n",
    "\n",
    "Similar to the one-hot encoding case, you can replace a value in a nominal column with the proportion of times that value appears in the column. For instance, if a column consists of the values `['a', 'b', 'a', 'c']`, then the proportion-encoded column is `[0.5, 0.25, 0.5, 0.25]`.  This might be a reasonable approach to predicting the price of a diamond, as you might expect *rarer attributes to be considered more valuable* than common ones.\n",
    "\n",
    "Complete the implementation of the function `create_proportions`, which takes in the `diamonds` DataFrame and returns a DataFrame of proportion-encoded features with names of the form `'proportion_<col>'`, where `'<col>'` is the original categorical column name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a205d92b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfc72abc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69d741d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# don't change this cell, but do run it -- it is needed for the tests to work\n",
    "diamonds = pd.read_csv(Path('data')/'diamonds.csv')\n",
    "out1_q3 = create_one_hot(diamonds)\n",
    "out2_q3 = create_proportions(diamonds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ea8bb4c",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b6d1048",
   "metadata": {},
   "source": [
    "### Question 4 – Quadratic Features 📈\n",
    "\n",
    "Linear regression doesn't capture non-linear relationships between variables. However, you can create features that encode such dependencies **before** fitting your regression model. Creating polynomial features is one way to do this. For example, the diamonds dataset contains `'x'`, `'y'`, and `'z'` dimensions for each stone. However, different combinations of size may be more valuable than others: a \"deep and wide\" diamond might be considered more valuable than a shallow, but \"long and wide\" diamond.\n",
    "\n",
    "Complete the implementation of the function `create_quadratics`, which takes in the `diamonds` DataFrame and returns a DataFrame of quadratic features of the form `'<col1> * <col2>'`, where `'<col1>'` and `'<col2>'` are the original quantitative columns. The output DataFrame should contain a column for every distinct pair of quantitative columns in `diamonds` (aside from `price`, which should be left out as it is what we are predicting). For instance, one of the columns in the returned DataFrame should named either `'carat * x'` or `'x * carat'`; the order of column names is not important.\n",
    "\n",
    "***Notes***:\n",
    "- Again, **do not** use `sklearn` for this question! \n",
    "- Try finding all pairs of quantitative columns efficiently; don't use a nested loop (hint: you may `import itertools`). Our solution contains just a single `for`-loop (over pairs of columns).\n",
    "- If you import itertools, you must include the itertools import in both this notebook and in `lab.py`.\n",
    "- The columns of the resulting DataFrame may be in any order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8815b67",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a364887d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7f39359",
   "metadata": {},
   "outputs": [],
   "source": [
    "# don't change this cell, but do run it -- it is needed for the tests to work\n",
    "diamonds = pd.read_csv(Path('data')/'diamonds.csv')\n",
    "out_q4 = create_quadratics(diamonds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cef3504",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "024a374f",
   "metadata": {},
   "source": [
    "### Question 5 – Comparing Performance 🏆\n",
    "\n",
    "We've now created several sets of features. **Which features are best able to predict the price of a diamond in a linear regression model?** We'll look at both single-feature linear regression models and multiple regression models. In all cases, use the default arguments to `sklearn`'s `LinearRegression` object (i.e. assume there is an intercept term).\n",
    "\n",
    "<br>\n",
    "\n",
    "Two of the evaluation metrics we can use to compare regression models are RMSE and $R^2$.\n",
    "\n",
    "RMSE, or Root Mean Squared Error, roughly measures the average distance between a model's predicted values and the actual values. The RMSE indicates how concentrated predictions are around the line of best fit, with lower values typically indicating smaller residuals and better model performance. RMSE can be calculated using the formula below, where $\\text{n}$ is the number of data points, $\\hat{y_i}$ is the $i$th prediction, and $y_i$ is the corresponding actual value.\n",
    "\n",
    "$$\\text{RMSE} = \\sqrt{\\frac{\\sum_{i=1}^n (y_i - \\hat{y_i})^2}{n}}$$\n",
    "\n",
    "Remember that in least squares regression, we choose the model parameters that minimize RMSE.\n",
    "\n",
    "<br>\n",
    "\n",
    "$R^2$, also known as the coefficient of determination, denotes the proportion of variance in the dependent variable that the independent variable explains. It is used to assess the quality of a linear fit, and in the simple linear regression case, it is the square of the correlation coefficient $r$. $R^2$ values range between 0 and 1, with lower values indicating a worse linear fit and with higher values indicating a better linear fit. If `model` is a fitted `sklearn` regression object, $R^2$ can be computed by calling `model.score`. \n",
    "\n",
    "#### Single-Feature Models\n",
    "\n",
    "- (1) Fit a single-feature linear regression model on `'carat'`. What is the $R^2$ of the model? (Note that `'carat'` turns out to be the best single feature to use in a linear model that predicts price.)\n",
    "- (2) What is the RMSE of the model you created in (1)?\n",
    "- (3) Amongst the other **quantitative** features present in the original `diamonds`, which produces the single-feature linear regression model with the highest $R^2$?\n",
    "- (4) Amongst all the new features you created in Questions 2-4, which produces the single-feature linear regression model with the highest $R^2$?\n",
    "- (5) Amongst the new categorical features you created in Question 2 and 3, which produces the single-feature linear regression model with the highest $R^2$? \n",
    "\n",
    "#### Multiple Regression\n",
    "\n",
    "Now, fit a multiple regression model using:\n",
    "- the quantitative columns that were present in the original `diamonds` dataset, and\n",
    "- the quantitative features engineered in Question 4\n",
    "\n",
    "as features. (Don't use any of the encodings of categorical columns from Questions 2 and 3.)\n",
    "\n",
    "- (6) What is the RMSE of this new model?\n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "Complete the implementation of the function `comparing_performance`, which returns a list containing the answers to the 6 questions numbered (1), (2), ..., (6) above. You don't need to round any of your answers.\n",
    "\n",
    "***Hint:*** Repeatedly use the `sklearn` pattern included below. It's a good idea to make a helper function that takes in a column, performs single-feature regression using the input column as the feature, and returns the $R^2$ and RMSE of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "929352b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "# X = ...\n",
    "# y = ...\n",
    "\n",
    "# lr = LinearRegression()\n",
    "# lr.fit(X, y)  # X is a DataFrame of training data; y is a Series of prices\n",
    "# lr.score(X, y)  # R-squared\n",
    "# lr.predict(X) # predicted prices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c23f00e8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cea22566",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5b60b0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# don't change this cell, but do run it -- it is needed for the tests to work\n",
    "import numbers\n",
    "out_q5 = comparing_performance()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8650c71f",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e13a16a7",
   "metadata": {},
   "source": [
    "## Part 3 – Feature Engineering with `sklearn` 🧠\n",
    "\n",
    "In this final question, you will use `sklearn`'s transformers and estimators for feature engineering. While everything you do with `sklearn` is possible to do with `pandas`, `sklearn` transformers enable you to couple your feature engineering with your modeling. This will allow you to more quickly build and assess your models in `sklearn`.\n",
    "\n",
    "Specifically, you will create a `TransformDiamonds` class that contains the three methods specified below – `transform_carat` (6.1), `transform_to_quantile` (6.2), and `transform_to_depth_pct` (6.3). In the starter code, there is a skeleton for `TransformDiamonds` that is initialized with a DataFrame `diamonds`.\n",
    "\n",
    "Each of the methods you implement in the `TransformDiamonds` class should take in a DataFrame, initialize a specific `sklearn.Transformer` object (like `Binarizer` or `FunctionTransformer`), and use the transformer to transform columns from the input DataFrame. You should **not** use DataFrame methods like `apply` in this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbcc52d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import Binarizer, QuantileTransformer, FunctionTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "459a06b8",
   "metadata": {},
   "source": [
    "Question 6 is made up of the three subparts below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "924d9258",
   "metadata": {},
   "source": [
    "### Question 6.1 – Transforming a Quantitative Column into a Binary Column (`transform_carat`)\n",
    "\n",
    "We call a diamond **large** if its weight is strictly greater than 1 carat. We want to **binarize** weights, so that they are 1 for large diamonds and 0 for small diamonds. Complete the implementation fo the method `transform_carat`, which takes in a DataFrame like `diamonds` and returns a binarized **array** of weights. Use a `Binarizer` object as your transformer.\n",
    "\n",
    "***Notes***:\n",
    "- You will return an array, not a Series, because `sklearn` thinks in terms of `np.ndarray`s, not DataFrames.\n",
    "- The implementation of this function should only take two lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c08514db",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a51d5f84",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d784739",
   "metadata": {},
   "outputs": [],
   "source": [
    "# don't change this cell, but do run it -- it is needed for the tests to work\n",
    "diamonds = pd.read_csv(Path('data')/'diamonds.csv')\n",
    "q6a_trans = TransformDiamonds(diamonds)\n",
    "q6a_out = q6a_trans.transform_carat(diamonds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8d8b37d",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q6.1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56393cb1",
   "metadata": {},
   "source": [
    "### Question 6.2 – Transforming a Quantitative Column into Quantiles (`transform_to_quantile`)\n",
    "\n",
    "You will now transform the `'carat'` column so that each diamond's weight in carats is replaced with the **percentile** amongst all diamonds in which its weight lies.\n",
    "\n",
    "Complete the implementation of the method `transform_to_quantiles`, which takes in a DataFrame like `diamonds` and returns an array containing the percentiles of the weight of each diamond, amongst all diamonds in `self.data`. This array should consist of proportions between 0 and 1; for instance, 0.65 will refer to the 65th percentile. The relevant transformer is `QuantileTransformer`. \n",
    "\n",
    "Some guidance:\n",
    "\n",
    "- Unlike with `Binarizer`, you need to `fit` your `QuantileTransformer` before calling `transform` on the input DataFrame `data`. \n",
    "    - You should `fit` your transformer on the DataFrame `self.data`, but you should only `transform` the `data` that is passed to `transform_to_quantiles`. \n",
    "    - Note that these two DataFrames, `self.data` and `data`, don't have to be the same! For instance, in the last two lines of the testing setup cell below, we fit a `QuantileTransformer` using just the first 1000 rows of `diamonds`, and then `transform` the entire `diamonds` DataFrame. Make sure your `transform_to_quantiles` method works in such a case.\n",
    "- When initializing your `QuantileTransformer`, use `n_quantiles=100`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55a713cd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76b7f68f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36f18f1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# don't change this cell, but do run it -- it is needed for the tests to work\n",
    "q6b_trans = TransformDiamonds(diamonds)\n",
    "q6b_out = q6b_trans.transform_to_quantile(diamonds)\n",
    "q6b_trans_top_1000 = TransformDiamonds(diamonds[:1000])\n",
    "q6b_out_top_1000 = q6b_trans_top_1000.transform_to_quantile(diamonds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74d484cd",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q6.2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f976186",
   "metadata": {},
   "source": [
    "### Question 6.3 – Transforming a Quantitative Column Using a Formula\n",
    "\n",
    "Recall from the introduction to Part 2 that the \"depth percentage\" of a diamond is defined as\n",
    "\n",
    "$$\\text{Depth Pct.} = 100\\% \\cdot \\frac{2z}{x + y}$$\n",
    "\n",
    "where $x$, $y$, and $z$ come from the `'x'`, `'y'`, and `'z'` columns in `diamonds`.\n",
    "\n",
    "Let's suppose that for some reason we don't have access to the `'depth'` column in `diamonds`, and instead need to recreate it just by looking at the `'x'`, `'y'`, and `'z'` columns. \n",
    "\n",
    "Complete the implementation of the method `transform_to_depth_pct`, which takes in a DataFrame like `diamonds` and returns an array consisting of the depth percentages of each diamond. Percentages should be between 0 and 100. The relevant transformer is `FunctionTransformer`.\n",
    "\n",
    "***Notes***:\n",
    "- To use `FunctionTransformer`, you will need to define your own function that takes in a 2D array and returns a single array.\n",
    "- Ignore `ZeroDivisionError` errors, and leave `np.NaN`s as is.\n",
    "- To verify your work, compare your outputted array to the actual `'depth'` column in `diamonds`.\n",
    "- It may seem like `FunctionTransformer` is totally unnecessary, since we can compute depth percentages using broadcasting directly. However, as we will see in lecture, transformers can be **pipelined** with other processing steps which greatly simplifies our code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6695780",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "949f0ac5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce894021",
   "metadata": {},
   "outputs": [],
   "source": [
    "# don't change this cell, but do run it -- it is needed for the tests to work\n",
    "diamonds = pd.read_csv(Path('data')/'diamonds.csv')\n",
    "q6c_trans = TransformDiamonds(diamonds)\n",
    "q6c_out = q6c_trans.transform_to_depth_pct(diamonds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c74423c",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"q6.3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e651b248",
   "metadata": {},
   "source": [
    "## Congratulations! You're done Lab 8! 🏁\n",
    "\n",
    "As a reminder, all of the work you want to submit needs to be in `lab.py`.\n",
    "\n",
    "To ensure that all of the work you want to submit is in `lab.py`, we've included a script named `lab-validation.py` in the lab folder. You shouldn't edit it, but instead, you should call it from the command line (e.g. the Terminal) to test your work.\n",
    "\n",
    "Once you've finished the lab, you should open the command line and run, in the directory for this lab:\n",
    "\n",
    "```\n",
    "python lab-validation.py\n",
    "```\n",
    "\n",
    "**This will run all of the `grader.check` cells that you see in this notebook, but only using the code in `lab.py` – that is, it doesn't look at any of the code in this notebook. If all of your `grader.check` cells pass in this notebook but not all of them pass in your command line with the above command, then you likely have code in your notebook that isn't in your `lab.py`!**\n",
    "\n",
    "You can also use `lab-validation.py` to test individual questions. For instance,\n",
    "\n",
    "```\n",
    "python lab-validation.py q1 q2 q4\n",
    "```\n",
    "\n",
    "will run the `grader.check` cells for Questions 1, 2, and 4 – again, only using the code in `lab.py`. [This video](https://www.loom.com/share/0ea254b85b2745e59322b5e5a8692e91?sid=5acc92e6-0dfe-4555-9b6a-8115b6a52f99) how to use the script as well.\n",
    "\n",
    "Once `python lab-validation.py` shows that you're passing all test cases, you're ready to submit your `lab.py` (and only your `lab.py`) to Gradescope. Once submitting to Gradescope, make sure to stick around until all test cases pass.\n",
    "\n",
    "There is also a call to `grader.check_all()` below in _this_ notebook, but make sure to also follow the steps above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a53752a",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "---\n",
    "\n",
    "To double-check your work, the cell below will rerun all of the autograder tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3262852e",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check_all()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  },
  "otter": {
   "tests": {
    "q1": {
     "name": "q1",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> best_transformation() in [1, 2, 3, 4]\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "q2": {
     "name": "q2",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> set(out_q2.columns) == {'ordinal_cut', 'ordinal_clarity', 'ordinal_color'}\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> np.unique(out_q2['ordinal_cut']).tolist() == [0, 1, 2, 3, 4]\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "q3": {
     "name": "q3",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> out1_q3.shape == (53940, 20)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> out1_q3.columns.str.startswith('one_hot').all()\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> out1_q3.isin([0,1]).all().all()\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> out2_q3.shape[1] == 3\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> out2_q3.columns.str.startswith('proportion_').all()\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> ((out2_q3 >= 0) & (out2_q3 <= 1)).all().all()\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "q4": {
     "name": "q4",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> out_q4.columns.str.contains(' * ').all()\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> ('x * z' in out_q4.columns) or ('z * x' in out_q4.columns)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> out_q4.shape[1] == 15\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "q5": {
     "name": "q5",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> len(out_q5) == 6\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> isinstance(out_q5[0], numbers.Real)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> all(isinstance(x, str) for x in out_q5[2:-1])\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        },
        {
         "code": ">>> out_q5[1] > out_q5[-1]\nTrue",
         "hidden": false,
         "locked": false,
         "points": 1
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "q6.1": {
     "name": "q6.1",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> isinstance(q6a_out, np.ndarray)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> q6a_out[172, 0] == 1\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> q6a_out[0, 0] == 0\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "q6.2": {
     "name": "q6.2",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> isinstance(q6b_out, np.ndarray)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> np.isclose(q6b_out[0, 0], 0.0075757, atol=0.0001)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> np.isclose(q6b_out[1, 0], 0.0025252, atol=0.0001)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    },
    "q6.3": {
     "name": "q6.3",
     "points": null,
     "suites": [
      {
       "cases": [
        {
         "code": ">>> len(q6c_out.shape) == 1\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        },
        {
         "code": ">>> np.isclose(q6c_out[0], 61.286, atol=0.0001)\nTrue",
         "hidden": false,
         "locked": false,
         "points": 0.5
        }
       ],
       "scored": true,
       "setup": "",
       "teardown": "",
       "type": "doctest"
      }
     ]
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}