PyStats-Solutions.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "143f0f81-3ddd-4215-8f34-814922b7c75a",
   "metadata": {},
   "source": [
    "# Python Statistics Essential Training"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "18badf4c-cd87-4f0a-b593-fb44ebb7a6e7",
   "metadata": {},
   "source": [
    "## Collecting and Cleaning Data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3611b204",
   "metadata": {},
   "source": [
    "### 01_01 - Loading data\n",
    "\n",
    "Using the Ames, Iowa Housing Data https://jse.amstat.org/v19n3/decock.pdf\n",
    "\n",
    "\n",
    "Goals:\n",
    "\n",
    "1. Load data from a CSV file using the `pd.read_csv` function.\n",
    "2. Understand how to access and interpret the shape of a DataFrame.\n",
    "3. Apply the `.describe` method to obtain summary statistics for a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "253b8681-0280-4fd8-8a9c-1116f37d2116",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "pd.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4814b9f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "url = 'https://github.com/mattharrison/datasets/raw/master/data/ames-housing-dataset.zip'\n",
    "url = 'data/ames-housing-dataset.zip'\n",
    "df = pd.read_csv(url, engine='pyarrow', dtype_backend='pyarrow')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c4b3b03-5812-40ef-81ac-1efc47c7110e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9c0838b-980a-4072-9045-25ec73ef1e0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7be747f7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "07a84328-1303-44b2-ade9-dcbdef948395",
   "metadata": {},
   "source": [
    "### 01_02 - Strings and Categories\n",
    "\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Understand the data types of columns in a DataFrame using the `.dtypes` attribute.\n",
    "* Select and filter categorical columns using the `.select_dtypes` method.\n",
    "* Compute and interpret summary statistics for categorical columns using the `.describe` method.\n",
    "* Determine the memory usage of string columns in a DataFrame.\n",
    "* Convert string columns to the `'category'` data type to save memory.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19f2b718",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78bb351f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Categoricals - Pandas 1.x\n",
    "df.select_dtypes(object)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d20fd4bc-3803-49d0-a722-7f61c583ed89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Categoricals - Pandas 2\n",
    "df.select_dtypes('string')  # or 'strings[pyarrow]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6de92880",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Categoricals\n",
    "df.select_dtypes('string').describe().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97035345",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .select_dtypes('string')\n",
    " .memory_usage(deep=True)\n",
    " .sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a74ee4f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .select_dtypes('string')\n",
    " .astype('category')\n",
    " .memory_usage(deep=True)\n",
    " .sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ed3c8e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "957_287 / 139_033"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f90840df",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d5d3047-cda1-4d12-a3b2-187c22083506",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Missing numeric columns (and strings in Pandas 1)\n",
    "(df\n",
    " .isna()\n",
    " .mean() \n",
    " .mul(100)\n",
    " .pipe(lambda ser: ser[ser > 0])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "233fd34f-dca0-4dc6-8b2c-426d011bd2dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Missing string values\n",
    "(df\n",
    " .select_dtypes('string')\n",
    " .eq('')\n",
    " .mean() \n",
    " .mul(100)\n",
    " .pipe(lambda ser: ser[ser > 0])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5a5ce84",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .select_dtypes('string')\n",
    " .eq('')\n",
    " .any(axis='columns')\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b496150e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# deleting\n",
    "# Notice Alley is NA!\n",
    "(df\n",
    " [~df.select_dtypes('string').eq('').any(axis='columns')]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16cc686a-2ca4-4022-b2f6-7aabd68a86a9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Looks like many values are NA\n",
    "# Missing string values (Encoded as NA)\n",
    "(df\n",
    " .select_dtypes('string')\n",
    " .eq('NA')\n",
    " .mean() \n",
    " .mul(100)\n",
    " .pipe(lambda ser: ser[ser > 0])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "540db4a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .query('`Pool QC`.isna()')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6639106-3840-4ffa-97dc-14cbc962fbb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .query('`Pool QC` == \"NA\"')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21a2b240",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Fill in empty string with 'Not Applicable'\n",
    "(df\n",
    " .assign(\n",
    "     **df.select_dtypes('string').replace('', 'Not Applicable'))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d34add5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examining unique values\n",
    "# Note the empty string\n",
    "(df\n",
    " .Electrical\n",
    " .value_counts()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "293c2160",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .query('Electrical == \"\"')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e56c2bf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This one was encoded as NA\n",
    "(df\n",
    " ['Fireplace Qu']\n",
    " .value_counts()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abaf75e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " ['Bsmt Cond']\n",
    " .value_counts()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31e15505",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting to Category\n",
    "(df\n",
    " .assign(\n",
    "     **df\n",
    "     .select_dtypes('string')\n",
    "     .replace('', 'Not Applicable')\n",
    "     .astype('category')\n",
    " )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7de8224f-5754-49dd-a442-de41da6b3972",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting to Category\n",
    "(df\n",
    " .assign(\n",
    "     **df\n",
    "     .select_dtypes('string')\n",
    "     .replace('', 'Not Applicable')\n",
    "     .astype('category')\n",
    " )\n",
    " .memory_usage(deep=True)\n",
    " .sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aba7b0e9-f9ff-42d4-a24f-0fd756cdbea0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting to Category\n",
    "(df\n",
    " .assign(\n",
    "     **df\n",
    "     .select_dtypes('string')\n",
    "     .replace('', 'Not Applicable')\n",
    "     #.astype('category')\n",
    " )\n",
    " .memory_usage(deep=True)\n",
    " .sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7fc320c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f1c2293",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7d9e9f62",
   "metadata": {},
   "source": [
    "### 01_03 - Cleaning Numbers\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Select and filter numeric columns using the `.select_dtypes` method.\n",
    "* Compute and interpret summary statistics for numeric columns using the `.describe` method.\n",
    "* Identify missing values in numeric columns.\n",
    "* Display a larger amount of data using options for minimum rows and maximum columns.\n",
    "* Utilize the `style` attribute to enhance the display of DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "485da6f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# In Pandas 1.x there would be many numbers here\n",
    "(df\n",
    " .select_dtypes(float)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "964a8437-e7a8-4280-a47a-9134d71acddb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(df\n",
    " .select_dtypes(int)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf9fafb2-ea5d-430b-a484-36feac07ee77",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(df\n",
    " .select_dtypes(int)\n",
    " .describe()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18501c48-1292-4f0b-b141-261edc5b5cb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c2df61f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(df\n",
    " .query('`Lot Frontage`.isna()')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f452323",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# How to see more data\n",
    "with pd.option_context('display.min_rows', 30, 'display.max_columns', 82):\n",
    "    display(df\n",
    "     .query('`Lot Frontage`.isna()')\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4af8102f-26a8-4ca6-add3-b9603dbabc44",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.style.set_sticky?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "216d6c9f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with pd.option_context('display.min_rows', 30, 'display.max_columns', 82):\n",
    "    display(df\n",
    "     .query('`Lot Frontage`.isna()')\n",
    "     .style\n",
    "     .set_sticky(axis='columns') # broken \n",
    "     .set_sticky(axis='index')\n",
    "    )    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6a4c7a3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Examine a column with missing values\n",
    "(df\n",
    " .query('`Garage Yr Blt`.isna()')\n",
    " )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd3e9bcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# missing + 2207!!!?\n",
    "(df\n",
    " ['Garage Yr Blt']\n",
    " .describe()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab319698",
   "metadata": {},
   "outputs": [],
   "source": [
    "# probably a typo!!\n",
    "with pd.option_context('display.min_rows', 30, 'display.max_columns', 82):  \n",
    "    display(df.query('`Garage Yr Blt` > 2200'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49c05832-d718-4bdd-87d6-2696a695140b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Any columns with Yr\n",
    "df.filter?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "544e3ed2-00c8-4231-9b43-22f39ce35bd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .filter(like='Yr')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "901f81ef-71c4-45f9-aeb1-d5a5b0c7b7eb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Any columns with Yr > 2023\n",
    "(df\n",
    " .filter(like='Yr')\n",
    " .pipe(lambda df_: df_[df_.gt(2023).any(axis='columns')])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa3b6d87-bb27-4b5b-89de-980457187c6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# What about \"Year\" columns?\n",
    "(df\n",
    " .rename(columns=lambda name: name.replace('Yr', 'Year'))\n",
    " .filter(like='Year')\n",
    " .pipe(lambda df_: df_[df_.gt(2023).any(axis='columns')])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28915611",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " ['Garage Yr Blt']\n",
    " .clip(upper=df['Year Built'].max())\n",
    " .value_counts()\n",
    " .sort_index()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74f3a253",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Update categories and clip\n",
    "# Inspect types\n",
    "(df\n",
    " .assign(**df.select_dtypes('string').replace('', 'Missing').astype('category'),\n",
    "         **{'Garage Yr Blt': df['Garage Yr Blt'].clip(upper=df['Year Built'].max())})\n",
    " .dtypes.value_counts()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "128918c9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9687d731",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "79222c4f",
   "metadata": {},
   "source": [
    "### 01_04 - Shrinking Numbers\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Create a function, `shrink_ints`, to automatically convert suitable integer columns to smaller integer types (`uint8`, `uint16`, `uint32`) based on their range of values.\n",
    "* Apply the `shrink_ints` function to the DataFrame to reduce memory usage while maintaining data integrity.\n",
    "* Create a function, `clean_housing`, that combines the data cleaning steps for string columns, clipping values in the \"Garage Yr Blt\" column, and shrinking integer columns.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79dc1c09-8aff-4423-ac8d-029a668d8392",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# continuing where we left off\n",
    "(df\n",
    " .assign(**df.select_dtypes('string').replace('', 'Missing').astype('category'),\n",
    "         **{'Garage Yr Blt': df['Garage Yr Blt'].clip(upper=df['Year Built'].max())})\n",
    " .describe()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3933ff70",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "for size in [np.uint8, np.uint16, np.uint32]:\n",
    "    print(np.iinfo(size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "211231f2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def shrink_ints(df):\n",
    "    mapping = {}\n",
    "    for col in df.dtypes[df.dtypes=='int64[pyarrow]'].index:\n",
    "        max_ = df[col].max()\n",
    "        min_ = df[col].min()\n",
    "        if min_ < 0:\n",
    "            continue\n",
    "        if max_ < 255:\n",
    "            mapping[col] = 'uint8[pyarrow]'\n",
    "        elif max_ < 65_535:\n",
    "            mapping[col] = 'uint16[pyarrow]'\n",
    "        elif max_ <  4294967295:\n",
    "            mapping[col] = 'uint32[pyarrow]'\n",
    "    return df.astype(mapping)\n",
    "            \n",
    "(df\n",
    " .assign(**df.select_dtypes('string').replace('', 'Missing').astype('category'),\n",
    "         **{'Garage Yr Blt': df['Garage Yr Blt'].clip(upper=df['Year Built'].max())})\n",
    " .pipe(shrink_ints)\n",
    " .describe()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9ab82e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .assign(**df.select_dtypes('string').replace('', 'Missing').astype('category'),\n",
    "         **{'Garage Yr Blt': df['Garage Yr Blt'].clip(upper=df['Year Built'].max())})\n",
    " .pipe(shrink_ints)\n",
    " .memory_usage(deep=True)\n",
    " .sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e7afb79",
   "metadata": {},
   "outputs": [],
   "source": [
    "(df\n",
    " .memory_usage(deep=True)\n",
    " .sum()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67ad5f59",
   "metadata": {},
   "outputs": [],
   "source": [
    "1_875_484 / 361_446"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75b9dee1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make function\n",
    "def shrink_ints(df):\n",
    "    mapping = {}\n",
    "    for col in df.dtypes[df.dtypes=='int64[pyarrow]'].index:\n",
    "        max_ = df[col].max()\n",
    "        min_ = df[col].min()\n",
    "        if min_ < 0:\n",
    "            continue\n",
    "        if max_ < 255:\n",
    "            mapping[col] = 'uint8[pyarrow]'\n",
    "        elif max_ < 65_535:\n",
    "            mapping[col] = 'uint16[pyarrow]'\n",
    "        elif max_ <  4294967295:\n",
    "            mapping[col] = 'uint32[pyarrow]'\n",
    "    return df.astype(mapping)\n",
    "\n",
    "\n",
    "def clean_housing(df):\n",
    "    return (df\n",
    "     .assign(**df.select_dtypes('string').replace('', 'Missing').astype('category'),\n",
    "             **{'Garage Yr Blt': df['Garage Yr Blt'].clip(upper=df['Year Built'].max())})\n",
    "     .pipe(shrink_ints)\n",
    "    )    \n",
    "\n",
    "clean_housing(df).dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dc36c2e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1734b042-1355-4009-a089-2f174d756337",
   "metadata": {},
   "source": [
    "### 01_05 - Challenge: Clean Ames \n",
    "\n",
    "* Create a cell containing all the imports for this notebook\n",
    "* Create a cell with the `clean_housing` and `shrink_ints` functions\n",
    "* Add code to load the raw data and create a `housing` variable from calling `clean_housing`\n",
    "* Move those cells to the top of the notebook\n",
    "* Restart the notebook and make sure that those cells work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f68f956",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 01_06\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93ee3cd2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca8fd274",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00e7e369",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8a1fbdb6",
   "metadata": {},
   "source": [
    "## Exploring & Visualizing"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "86e54be6-bad3-4533-86fe-5aa8370ed81a",
   "metadata": {},
   "source": [
    "### 02_01 - Categorical Exploration\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Explore a categorical column, such as \"MS Zoning,\" by accessing the column and displaying its unique values.\n",
    "* Visualize the value counts of a categorical column using a bar chart.\n",
    "* Visualize the value counts of a categorical column using a horizontal bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "923e7b82",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "url = 'data/ames-housing-dataset.zip'\n",
    "raw = pd.read_csv(url, engine='pyarrow', dtype_backend='pyarrow')\n",
    "\n",
    "# make function\n",
    "def shrink_ints(df):\n",
    "    mapping = {}\n",
    "    for col in df.dtypes[df.dtypes=='int64[pyarrow]'].index:\n",
    "        max_ = df[col].max()\n",
    "        min_ = df[col].min()\n",
    "        if min_ < 0:\n",
    "            continue\n",
    "        if max_ < 255:\n",
    "            mapping[col] = 'uint8[pyarrow]'\n",
    "        elif max_ < 65_535:\n",
    "            mapping[col] = 'uint16[pyarrow]'\n",
    "        elif max_ <  4294967295:\n",
    "            mapping[col] = 'uint32[pyarrow]'\n",
    "    return df.astype(mapping)\n",
    "\n",
    "\n",
    "def clean_housing(df):\n",
    "    return (df\n",
    "     .assign(**df.select_dtypes('string').replace('', 'Missing').astype('category'),\n",
    "             **{'Garage Yr Blt': df['Garage Yr Blt'].clip(upper=df['Year Built'].max())})\n",
    "     .pipe(shrink_ints)\n",
    "    )    \n",
    "\n",
    "housing = clean_housing(raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf05c2bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "housing.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da9b0f2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# categoricals\n",
    "(housing\n",
    "  ['MS Zoning'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b064c0c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# categoricals\n",
    "(housing\n",
    "  ['MS Zoning']\n",
    "  .value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "467ed1c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# categoricals\n",
    "(housing\n",
    "  ['MS Zoning']\n",
    "  .value_counts()\n",
    "  .plot.bar())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97f51b33",
   "metadata": {},
   "outputs": [],
   "source": [
    "# categoricals\n",
    "(housing\n",
    "  ['MS Zoning']\n",
    "  .value_counts()\n",
    "  .plot.barh())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a38744a5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67bc9e09",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "76798903",
   "metadata": {},
   "source": [
    "### 02_02: Histograms and Distributions\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Obtain descriptive statistics of the \"SalePrice\" column using the `.describe` method.\n",
    "* Visualize the distribution of the \"SalePrice\" column using a histogram.\n",
    "* Customize the histogram by specifying the number of bins using the `bins` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5a83622",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numerical\n",
    "(housing\n",
    " .SalePrice\n",
    " .describe()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4349dd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numerical\n",
    "(housing\n",
    " .SalePrice\n",
    " .hist()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4205492",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numerical\n",
    "(housing\n",
    " .SalePrice\n",
    " .hist(bins=130)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b61602fe",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67026188",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "14f6ee14",
   "metadata": {},
   "source": [
    "### 02_03 - Outliers and Z-scores\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Calculate the Z-score for the \"SalePrice\" column using the `calc_z` function.\n",
    "* Identify outliers based on the Z-score by assigning a boolean column indicating whether the Z-score is greater than or equal to 3 or less than or equal to -3.\n",
    "* Identify outliers using the IQR (interquartile range) method by assigning a boolean column indicating whether the values are outside the range of median ± 3 * IQR.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6249a7e4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# outlier with Z-score\n",
    "def calc_z(df, col):\n",
    "    mean = df[col].mean() \n",
    "    std = df[col].std()\n",
    "    return (df[col]-mean)/std\n",
    "\n",
    "(housing\n",
    " .pipe(calc_z, col='SalePrice')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d6b29b4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(housing\n",
    " .assign(z_score=calc_z(housing, col='SalePrice'))\n",
    " #.query('z_score.abs() >= 3')\n",
    " .query('z_score <= -3')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae7c2e8a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def calc_iqr_outlier(df, col):\n",
    "    ser = df[col]\n",
    "    iqr = ser.quantile(.75) - ser.quantile(.25)\n",
    "    med = ser.median()\n",
    "    small_mask = ser < med-iqr*3\n",
    "    large_mask = ser > med+iqr*3\n",
    "    return small_mask | large_mask\n",
    "\n",
    "housing[\n",
    "calc_iqr_outlier(housing, 'SalePrice')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a430cbea-cd55-4046-93a5-908e7fe11218",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def calc_iqr_outlier(df, col):\n",
    "    ser = df[col]\n",
    "    iqr = ser.quantile(.75) - ser.quantile(.25)\n",
    "    med = ser.median()\n",
    "    small_mask = ser < med-iqr*3\n",
    "    large_mask = ser > med+iqr*3\n",
    "    return small_mask | large_mask\n",
    "\n",
    "(housing\n",
    " .assign(iqr_outlier=calc_iqr_outlier(housing, col='SalePrice'))\n",
    " .query('iqr_outlier')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad9daf1d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c235ff1d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c4ba172c",
   "metadata": {},
   "source": [
    "### 02_04 - Correlations\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Calculate the Pearson correlation\n",
    "* Calculate the Spearman correlation \n",
    "* Color a correlation matrix appropriately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b54bceb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Pearson correlation\n",
    "housing.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e3d4f5e-5076-46b7-a037-383971c6ae17",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "housing.corr(numeric_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce4c9263",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(housing\n",
    " .corr(method='spearman', numeric_only=True)\n",
    " .style\n",
    " .background_gradient()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68041c0f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(housing\n",
    " .corr(method='spearman', numeric_only=True)\n",
    " .style\n",
    " .background_gradient(cmap='RdBu')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3443cb16-2a77-4694-8ec4-5d8590387648",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(housing\n",
    " .corr(method='spearman', numeric_only=True)\n",
    " .style\n",
    " .background_gradient(cmap='RdBu', vmin=-1, vmax=1)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33f643df",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec490403",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "783f0620",
   "metadata": {},
   "source": [
    "### 02_05 - Scatter Plots\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Create a scatter plot\n",
    "* Set transparency\n",
    "* Jitter plot values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b7d78a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "(housing\n",
    " .plot\n",
    " .scatter(x='Year Built', y='Overall Cond')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e12046f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "housing['Year Built'].corr(housing['Overall Cond'], method='spearman')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8380db69",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(housing\n",
    " .plot\n",
    " .scatter(x='Year Built', y='Overall Cond', alpha=.1)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe1f0ac8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# with jitter in y\n",
    "(housing\n",
    " .assign(**{'Overall Cond': housing['Overall Cond'] + np.random.random(len(housing))*.8 -.4})\n",
    " .plot\n",
    " .scatter(x='Year Built', y='Overall Cond', alpha=.1)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61c778a1-b3ac-41eb-9e6c-78917bdee407",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make function\n",
    "def jitter(df_, col, amount=.5):\n",
    "    return (df_\n",
    "            [col] + np.random.random(len(df_))*amount - (amount/2))\n",
    "    \n",
    "(housing\n",
    " .assign(#**{'Overall Cond': housing['Overall Cond'] + np.random.random(len(housing))*.8 -.4})\n",
    "     **{'Overall Cond': jitter(housing, 'Overall Cond', amount=.8)})\n",
    " .plot\n",
    " .scatter(x='Year Built', y='Overall Cond', alpha=.1)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49ad3eec",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\n",
    "(housing\n",
    " #.assign(**{'Overall Cond': housing['Overall Cond'] + np.random.random(len(housing))*.8 -.4})\n",
    " .plot\n",
    " .hexbin(x='Year Built', y='Overall Cond', alpha=1, gridsize=18)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2af3653d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69c5720d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cafa7585",
   "metadata": {},
   "source": [
    "### 02_06 - Visualizing Categoricals and Numerical Values\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Create a box plot of a single column\n",
    "* Create a box plot of multiple columns\n",
    "* Use the `.pivot` method\n",
    "* Use Seaborn to create other distibution plots by category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b514b6a3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Numerical and categorical\n",
    "(housing\n",
    " #.assign(**{'Overall Cond': housing['Overall Cond'] + np.random.random(len(housing))*.8 -.4})\n",
    " .plot\n",
    " .box(x='Year Built', y='Overall Cond')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a812e16",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make multiple box plots\n",
    "(housing\n",
    " .pivot(columns='Year Built', values='Overall Cond')\n",
    " .apply(lambda ser: ser[~ser.isna()].reset_index(drop=True))\n",
    " .plot.box()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4b957d0-6398-4908-90bf-8bb5aaee372e",
   "metadata": {},
   "outputs": [],
   "source": [
    "(housing\n",
    " .pivot(columns='Year Built', values='Overall Cond')\n",
    " .apply(lambda ser: ser[~ser.isna()].reset_index(drop=True))\n",
    " .loc[:, [1900, 1920, 1940, 1960, 1980, 2000]]\n",
    " .plot.box()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d13b103",
   "metadata": {},
   "outputs": [],
   "source": [
    "1993 // 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71af5b47-fc69-431c-aff4-2eb124c478ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Group by decade\n",
    "(housing\n",
    " .assign(decade=(housing['Year Built']//10 ) * 10)\n",
    " .pivot(columns='decade', values='Overall Cond')\n",
    " .apply(lambda ser: ser[~ser.isna()].reset_index(drop=True))\n",
    " .plot.box()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e7d217f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# or use seaborn\n",
    "import seaborn as sns\n",
    "\n",
    "sns.boxplot(data=housing, x='Year Built', y='Overall Cond')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27dfe107-1ef9-47a0-b6ba-8b5b54b8d14f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sns.boxplot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdf512c6-2ef1-4c25-b5c2-dbeccaf4e120",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sns.boxplot(data=housing, x='Year Built', y='Overall Cond',\n",
    "            order=[1900, 1920, 1940]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dfa3b8b-b679-4090-aaa4-dcfd9c674406",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sns.violinplot(data=housing, x='Year Built', y='Overall Cond',\n",
    "            order=[1900, 1920, 1940]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d148bea-aed2-4e2b-b827-248a2512a14a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sns.boxenplot(data=housing, x='Year Built', y='Overall Cond',\n",
    "            order=[1900, 1920, 1940]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d745c1e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4bdda68",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0bf01039",
   "metadata": {},
   "source": [
    "### 02_07 - Comparing Two Categoricals\n",
    "\n",
    "Goals:\n",
    "\n",
    "\n",
    "* Create a cross-tabulation \n",
    "* Style the cross-tabulation table \n",
    "* Explore the documentation of the `.background_gradient` method of pandas styling.\n",
    "* Create a stacked bar plot of a cross-tabulation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73f62cd5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# 2 Categoricals\n",
    "housing.dtypes[:40]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9f6aa2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2 Categoricals - Cross tabulation\n",
    "(housing\n",
    " .groupby(['Overall Qual', 'Bsmt Cond'])\n",
    " .size()\n",
    " .unstack()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ee3ff6a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(pd.crosstab(index=housing['Overall Qual'], columns=housing['Bsmt Cond']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2a6cb11",
   "metadata": {},
   "outputs": [],
   "source": [
    "(pd.crosstab(index=housing['Overall Qual'], columns=housing['Bsmt Cond'])\n",
    " .style\n",
    " .background_gradient(cmap='viridis', axis=None)  # None is whole dataframe\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bd90440",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw.style.background_gradient?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "642800e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reorder\n",
    "# Ex: Excellent\n",
    "# Gd: Good\n",
    "# TA: Typical - slight dampness allowed\n",
    "# Fa: Fair - dampness or some cracking or settling\n",
    "# Po: Poor - Severe cracking, settling, or wetness\n",
    "    \n",
    "(pd.crosstab(index=housing['Overall Qual'], columns=housing['Bsmt Cond'])\n",
    " .loc[:, ['Ex', 'Gd', 'TA', 'Fa', 'Po', 'Missing', 'NA']]\n",
    " .style\n",
    " .background_gradient(cmap='viridis', axis=None)  # None is whole dataframe\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61c962da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reorder\n",
    "# Ex: Excellent\n",
    "# Gd: Good\n",
    "# TA: Typical - slight dampness allowed\n",
    "# Fa: Fair - dampness or some cracking or settling\n",
    "# Po: Poor - Severe cracking, settling, or wetness\n",
    "    \n",
    "(pd.crosstab(index=housing['Overall Qual'], columns=housing['Bsmt Cond'])\n",
    " .loc[:, ['Ex', 'Gd', 'TA', 'Fa', 'Po', 'Missing', 'NA']]\n",
    " .plot.bar(stacked=True, \n",
    "           cmap='viridis')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6722e1a3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "36795045-b07c-45a3-9e44-3be72e46cfbd",
   "metadata": {},
   "source": [
    "### 02_08 - Challenge: Explore Ames\n",
    "* Create a scatter plot of *1st Flr SF* against *SalePrice*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f718cfe-927f-41db-b42f-cbe992d5c151",
   "metadata": {},
   "outputs": [],
   "source": [
    "#02_09 - Solution\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1424b2fe",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "509832ba",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5971a279",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a658dbb5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6fd87c4f",
   "metadata": {},
   "source": [
    "## Linear Regression\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ac76f953-f772-449a-b142-f83b6a3b144d",
   "metadata": {},
   "source": [
    "### 03_01 - Linear Regression\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Understand how to prepare data for linear regression by selecting the relevant numerical features and the target variable.\n",
    "* Learn to split the data into training and testing sets using `train_test_split` \n",
    "* Evaluate the performance of the linear regression model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e19599d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model, model_selection, preprocessing\n",
    "\n",
    "X = housing.select_dtypes('number').drop(columns='SalePrice')\n",
    "y = housing.SalePrice\n",
    "\n",
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1e4be14",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68b87038",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0165a7a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "lr = linear_model.LinearRegression()\n",
    "lr.fit(X_train, y_train)\n",
    "lr.score(X_test, y_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b435944c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "X_train.isna().any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56e93237-3f6b-4033-9a42-86708c6529b1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "url = 'data/ames-housing-dataset.zip'\n",
    "raw = pd.read_csv(url, engine='pyarrow', dtype_backend='pyarrow')\n",
    "\n",
    "# make function\n",
    "def shrink_ints(df):\n",
    "    mapping = {}\n",
    "    for col in df.dtypes[df.dtypes=='int64[pyarrow]'].index:\n",
    "        max_ = df[col].max()\n",
    "        min_ = df[col].min()\n",
    "        if min_ < 0:\n",
    "            continue\n",
    "        if max_ < 255:\n",
    "            mapping[col] = 'uint8[pyarrow]'\n",
    "        elif max_ < 65_535:\n",
    "            mapping[col] = 'uint16[pyarrow]'\n",
    "        elif max_ <  4294967295:\n",
    "            mapping[col] = 'uint32[pyarrow]'\n",
    "    return df.astype(mapping)\n",
    "\n",
    "\n",
    "def clean_housing_no_na(df):\n",
    "    return (df\n",
    "     .assign(**df.select_dtypes('string').replace('', 'Missing').astype('category'),\n",
    "             **{'Garage Yr Blt': df['Garage Yr Blt'].clip(upper=df['Year Built'].max())})\n",
    "     .pipe(shrink_ints)\n",
    "     .pipe(lambda df_: df_.assign(**df_.select_dtypes('number').fillna(0)))\n",
    "    )    \n",
    "    \n",
    "\n",
    "housing2 = clean_housing_no_na(raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d1a0579",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = housing2.select_dtypes('number').drop(columns='SalePrice')\n",
    "y = housing2.SalePrice\n",
    "\n",
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bea311b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = linear_model.LinearRegression()\n",
    "lr.fit(X_train, y_train)\n",
    "lr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "391ca314",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36ba2812",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "88b0d89f",
   "metadata": {
    "incorrectly_encoded_metadata": "jp-MarkdownHeadingCollapsed=true"
   },
   "source": [
    "### 03_02 - Interpreting Linear Regression Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d12e7618",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "lr.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aa4ba57",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.intercept_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2d91acc",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr.feature_names_in_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf7f7ab0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.Series(lr.coef_, index=lr.feature_names_in_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "309e6674",
   "metadata": {},
   "outputs": [],
   "source": [
    "(pd.Series(lr.coef_, index=lr.feature_names_in_)\n",
    " .sort_values()\n",
    " .plot.barh())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b78379f4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(pd.Series(lr.coef_, index=lr.feature_names_in_)\n",
    " .pipe(lambda ser: ser[ser.abs() > 100])\n",
    " .sort_values()\n",
    " .plot.barh())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "789e7322-1686-4bb9-a487-9eabfad2c817",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(pd.Series(lr.coef_, index=lr.feature_names_in_)\n",
    " .pipe(lambda ser: ser[ser.abs() > 100])\n",
    " .sort_values()\n",
    " .index\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abfb3ae3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40c5fee8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "525eda5f",
   "metadata": {},
   "source": [
    "### 03_03 - Standardizing Values\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Understand the concept of standardization \n",
    "* Learn to use the `StandardScaler` class \n",
    "* Evaluate the impact of standardization on the performance of the linear regression model using the coefficient of determination (R-squared) score.\n",
    "* Visualize the coefficients of the linear regression model using a horizontal bar plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b845657",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = housing2.select_dtypes('number').drop(columns='SalePrice')\n",
    "y = housing2.SalePrice\n",
    "\n",
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, random_state=42)\n",
    "\n",
    "std = preprocessing.StandardScaler()\n",
    "X_train = std.fit_transform(X_train)\n",
    "X_test = std.transform(X_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e41ee6ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# was .84\n",
    "lr = linear_model.LinearRegression()\n",
    "lr.fit(X_train, y_train)\n",
    "lr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e73fc46f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(pd.Series(lr.coef_, index=X.columns)\n",
    " .sort_values()\n",
    " .plot.barh())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bf483dc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(pd.Series(lr.coef_, index=X.columns)\n",
    " .sort_values()\n",
    "# .plot.barh()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0632ab4b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(pd.Series(lr.coef_, index=X.columns)\n",
    " .sort_values()\n",
    " .pipe(lambda ser: ser[ser.abs() > 1e8])\n",
    " .plot.barh()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df631b4d-2424-42f0-aba0-1f3b7aeb03f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "(pd.Series(lr.coef_, index=X.columns)\n",
    " .sort_values()\n",
    " .pipe(lambda ser: ser[ser.abs() > 1e8])\n",
    " .index\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddfcbd1c-b0cf-4b03-8603-eff5ae182e50",
   "metadata": {},
   "outputs": [],
   "source": [
    "simple_feats = set(['Kitchen AbvGr', 'Yr Sold', 'Bedroom AbvGr', 'Half Bath',\n",
    "       'Bsmt Half Bath', 'MS SubClass', 'Full Bath', 'Year Remod/Add',\n",
    "       'Mo Sold', 'Year Built', 'TotRms AbvGrd', 'Fireplaces', 'Overall Cond',\n",
    "       'Bsmt Full Bath', 'Garage Cars', 'Overall Qual'])\n",
    "std_feats = set(['Total Bsmt SF', '2nd Flr SF', '1st Flr SF', 'Low Qual Fin SF',\n",
    "       'BsmtFin SF 2', 'Gr Liv Area', 'Bsmt Unf SF', 'BsmtFin SF 1'])\n",
    "print(sorted(simple_feats | std_feats))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36e74d24",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Look at correlations \n",
    "(X\n",
    " .assign(SalePrice=y)\n",
    " .corr()\n",
    " .loc[['SalePrice', '1st Flr SF', '2nd Flr SF', 'Bedroom AbvGr', 'Bsmt Full Bath', 'Bsmt Half Bath', \n",
    "       'Bsmt Unf SF', 'BsmtFin SF 1', 'BsmtFin SF 2', 'Fireplaces', 'Full Bath', \n",
    "       'Garage Cars', 'Gr Liv Area', 'Half Bath', 'Kitchen AbvGr', 'Low Qual Fin SF',\n",
    "       'MS SubClass', 'Mo Sold', 'Overall Cond', 'Overall Qual', 'TotRms AbvGrd', \n",
    "       'Total Bsmt SF', 'Year Built', 'Year Remod/Add', 'Yr Sold']]\n",
    " .style\n",
    " .background_gradient(cmap='RdBu', vmin=-1, vmax=1)\n",
    " .set_sticky(axis='index') \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "067d36b8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "47ee52cc",
   "metadata": {},
   "source": [
    "### 03_04 - Regression with XGBoost\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Learn to use the XGBoost library (`xgboost`) for regression tasks.\n",
    "* Evaluate the performance of the XGBoost model.\n",
    "* Explore the importance of features in the XGBoost model using the feature importances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a7e9d0f-e3f3-491c-8136-448e8ecd572c",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = housing2.select_dtypes('number').drop(columns='SalePrice')\n",
    "y = housing2.SalePrice\n",
    "\n",
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, random_state=42)\n",
    "\n",
    "std = preprocessing.StandardScaler().set_output(transform='pandas')\n",
    "X_train = std.fit_transform(X_train)\n",
    "X_test = std.transform(X_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cba22f5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "# was .84\n",
    "xg = xgb.XGBRegressor()\n",
    "xg.fit(X_train, y_train)\n",
    "xg.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0841e693-2111-4098-badc-01b220bdd0a7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.Series(xg.feature_importances_, index=X_train.columns).sort_values().plot.barh(figsize=(3,12))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6de0e46c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Use categories\n",
    "# (Need to convert pyarrow numbers to pandas 1.x numbers)\n",
    "X_cat = (housing.assign(**housing.select_dtypes('number').astype('Int64')).drop(columns='SalePrice'))\n",
    "\n",
    "y_cat = housing.SalePrice\n",
    "X_cat_train, X_cat_test, y_cat_train, y_cat_test = model_selection.train_test_split(X_cat, y_cat, random_state=42)\n",
    "xg_cat = xgb.XGBRegressor(enable_categorical=True, tree_method='hist')\n",
    "xg_cat.fit(X_cat_train, y_cat_train)\n",
    "xg_cat.score(X_cat_test, y_cat_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef6d5f39",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pd.Series(xg_cat.feature_importances_, index=xg_cat.feature_names_in_).sort_values().plot.barh(figsize=(3,12))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a40ee7f0-685a-428a-907f-f3aac0f9f49d",
   "metadata": {},
   "source": [
    "### 03_05 - Challenge: Predict Ames\n",
    "* Create a linear regression model using the top 5 features from the (non-categorical) XGBoost model\n",
    "* What is the `.score` of the model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53c54159-97be-457a-a843-a8473624bafc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 03_06\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57a354f5-46a4-4384-8942-8e6e28efa6a2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6280198-36dd-4c27-b6fa-3efa4f0fb300",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03bf0b2a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7549c85f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4283e848",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b921a545",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fc7d81ea-fe36-49ad-a918-0ac1c8807845",
   "metadata": {},
   "source": [
    "## Hypothesis Test"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "55d459c7-2e8b-4a5c-9800-ab9175e4a74f",
   "metadata": {},
   "source": [
    "### 04_01 - Exploring Data\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Explore summary statistics by group\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fee9996d-ae98-433b-a68f-4d4c2d4c1ee2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "housing.Neighborhood.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d23a73ea-33ff-4032-8ef5-5725f99297b4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(housing\n",
    " .groupby('Neighborhood')\n",
    " .describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab4c2bd7-35f7-4bfd-91c3-4066dba6d0f3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(housing\n",
    " .groupby('Neighborhood')\n",
    " .describe()\n",
    " .loc[['CollgCr', 'NAmes'], ['SalePrice']]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "821f1ff0-9e6c-4f7f-909c-377970221953",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "(housing\n",
    " .groupby('Neighborhood')\n",
    " .describe()\n",
    " .loc[['CollgCr', 'NAmes'], ['SalePrice']]\n",
    " .T\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1402761d-55b1-4a5a-9127-00a98bffa4cf",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2d05050-8d8e-415c-aa6f-0cc93db53563",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c7a5d72-d8f0-4d8d-967d-e24410c54b02",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cfa2a8d9-3018-4f9b-9844-69be929c0c85",
   "metadata": {},
   "source": [
    "### 04_02 - Visualizing Distributions\n",
    "\n",
    "Goals\n",
    "\n",
    "* Make histograms of both distributions\n",
    "* Make a cumulative distribution plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44390c28-52d5-4e59-ba35-446a39d0c708",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_ames = (housing\n",
    "          .query('Neighborhood == \"NAmes\"')\n",
    "          .SalePrice)\n",
    "college_cr = (housing\n",
    "          .query('Neighborhood == \"CollgCr\"')\n",
    "          .SalePrice)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8a04bd0-4cc9-4903-a59c-0fc5de6230ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = n_ames.hist(label='NAmes')\n",
    "college_cr.hist(ax=ax, label='CollgCr')\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c8ad564-3960-4b7a-a8d1-3dad944ca8dc",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "alpha = .7\n",
    "ax = n_ames.hist(label='NAmes', alpha=alpha)\n",
    "college_cr.hist(ax=ax, label='CollgCr', alpha=alpha)\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d3f7af2-8fb7-4e9e-a87b-9f29b91e6514",
   "metadata": {},
   "outputs": [],
   "source": [
    "(n_ames\n",
    " .to_frame()\n",
    " .assign(cdf=n_ames.rank(method='average', pct=True))\n",
    " .sort_values(by='SalePrice')\n",
    " .plot(x='SalePrice', y='cdf', label='NAmes')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d817f383-bf0c-4426-8b65-37b987178037",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_cdf(ser, ax=None, label=''):\n",
    "    (ser\n",
    "     .to_frame()\n",
    "     .assign(cdf=ser.rank(method='average', pct=True))\n",
    "     .sort_values(by='SalePrice')\n",
    "     .plot(x='SalePrice', y='cdf', label=label, ax=ax)\n",
    "    )\n",
    "    return ser\n",
    "plot_cdf(n_ames, label='NAmes')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1bdbaf5-a508-4558-870c-54b2c45bf013",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def plot_cdf(ser, ax=None, label=''):\n",
    "    (ser\n",
    "     .to_frame()\n",
    "     .assign(cdf=ser.rank(method='average', pct=True))\n",
    "     .sort_values(by='SalePrice')\n",
    "     .plot(x='SalePrice', y='cdf', label=label, ax=ax)\n",
    "    )\n",
    "    return ser\n",
    "    \n",
    "fig, ax = plt.subplots(figsize=(8,4))\n",
    "plot_cdf(n_ames, label='NAmes', ax=ax)\n",
    "plot_cdf(college_cr, label='CollegeCr', ax=ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3028a7cc-f542-4cee-94fa-70582a393e80",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e638572-c5ff-4ad6-9e90-4ec954a87634",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3673e9a8-2f6e-43ee-97d9-fcd73d0c73cf",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3820c631-64ca-41b4-8d84-a9ff5db35c9c",
   "metadata": {},
   "source": [
    "### 04_03 - Running Statistical Tests\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Use the `scipy.stats` module to run a statistical test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80157760-1b37-422c-b7cb-d616c95117ff",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(dir(stats))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "184fca4b-df25-4ba8-8cbc-078290207ff1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "stats.ks_2samp?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfc25b0d-49c1-4cc7-8395-7ba3e2f5f24d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ks_statistic, p_value = stats.ks_2samp(n_ames, college_cr)\n",
    "print(ks_statistic, p_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d93a6999-3ccb-4ba5-81b2-810c103e7728",
   "metadata": {},
   "outputs": [],
   "source": [
    "if p_value > 0.05:\n",
    "    print('Fail to reject null hypothesis: Same distribution')\n",
    "else:\n",
    "    print('Reject null hypothesis: Not from the same distribution')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa2cbfc0-682b-4d2b-9264-43d6149d8d7a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "105ea651-1070-4cc9-9cf8-8b88738f7cf8",
   "metadata": {},
   "source": [
    "### 04_04 - Testing for Normality\n",
    "\n",
    "Goals:\n",
    "\n",
    "* Use the `scipy.stats` module to test for normality\n",
    "* Use the `scipy.stats` module to create a probability plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74dd1b84-7605-45d0-b06b-2066e372c083",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the Shapiro-Wilks test\n",
    "shapiro_stat, p_value = stats.shapiro(n_ames)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c8e8f6f-cb5d-4c78-b002-d0c1ea3c4116",
   "metadata": {},
   "outputs": [],
   "source": [
    "if p_value > 0.05:\n",
    "    print(\"The distribution of the series is likely normal (fail to reject H0)\")\n",
    "else:\n",
    "    print(\"The distribution of the series is likely not normal (reject H0)\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1d995f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "p_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a17f111-b611-47e3-a847-358ff92fbb3d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "stats.probplot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bcd1ac6-fea6-47f5-84a1-0c9c892738ce",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8,4))\n",
    "_ = stats.probplot(n_ames, plot=ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30527c58-d1ee-408c-8dcb-e398bb326f42",
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = .7\n",
    "ax = n_ames.hist(label='NAmes', alpha=alpha)\n",
    "college_cr.hist(ax=ax, label='CollgCr', alpha=alpha)\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af6975e4-5788-48b3-a2e7-97f91897babf",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8,4))\n",
    "_ = stats.probplot(college_cr, plot=ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf671486-5bcc-45ee-8277-7c6b5e169a37",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21071784-bafd-436f-96ed-a14eec1029c4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "89af64b6-8c19-41f9-9d68-fdae8b93ca34",
   "metadata": {},
   "source": [
    "### 04_05 - Challenge: Checking Square Footage Distributions\n",
    "* Is the distribution of *1st Flr SF* from *NAmes* and *CollgCr* the same?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87d495ab-1f37-495e-bbd5-506a8e191164",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 04_06\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dfa41dd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "292fa585-93f2-49c0-9545-685b03290f3e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2cac26c-c687-44eb-ab80-8b4fb140985a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d479a838",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecf2ffed",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b8a8cef",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}