1.-Data Cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Descriptive Statistics Review"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before you start:\n",
    "\n",
    "- Read the README.md file\n",
    "- Comment as much as you can\n",
    "- Happy learning!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](./diamonds.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we are going to work with data to understand the characteristics of a diamond that are most likely to influence its price. In this first part of the lab, we will explore and clean our data. \n",
    "\n",
    "The dataset we will be using is comprised of approximately 54k rows and 11 different columns. As always, a row represents a single observation (in this case a diamond) and each of the columns represent a different feature of a diamond.\n",
    "\n",
    "The following codebook was provided together with the dataset to clarify what each column represents:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Column  | Description  |\n",
    "|---|---|\n",
    "| Price  | Price in US dollars (326-18,823)  |\n",
    "| Carat  | Weight of the diamond (0.2--5.01)  |\n",
    "| Cut  | Quality of the cut (Fair, Good, Very Good, Premium, Ideal)  |\n",
    "| Color  | Diamond colour, from J (worst) to D (best)  |\n",
    "| Clarity  | A measurement of how clear the diamond is (I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best))   |\n",
    "| x  | Length in mm (0--10.74)  |\n",
    "| y  | Width in mm (0--58.9)  |\n",
    "| z  | Depth in mm (0--31.8)  |\n",
    "| Depth  | Total depth percentage = z / mean(x, y) = 2 * z / (x + y) (43--79)  |\n",
    "| Table  | Width of top of diamond relative to widest point (43--95)  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Libraries\n",
    "Pandas and numpy will be needed for the analysis of the data. Don't worry about the seaborn and matplotlib import at the moment, you will learn more about them next week, but we will be using some of their functionalities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First import the data from the .csv file provided and assign it to a variable named `diamonds` and drop the column with the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n",
    "diamonds = pd.read_csv('diamonds.csv')\n",
    "diamonds = diamonds.drop('Unnamed: 0', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Taking the first look at the data.\n",
    "Let's see how the data looks by using pandas methods like `head()`, `info()` and `describe()`. \n",
    "\n",
    "**First, use the `head` method.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.5</td>\n",
       "      <td>55.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.95</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2.43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>Premium</td>\n",
       "      <td>E</td>\n",
       "      <td>SI1</td>\n",
       "      <td>59.8</td>\n",
       "      <td>61.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.89</td>\n",
       "      <td>3.84</td>\n",
       "      <td>2.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Good</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>56.9</td>\n",
       "      <td>65.0</td>\n",
       "      <td>327</td>\n",
       "      <td>4.05</td>\n",
       "      <td>4.07</td>\n",
       "      <td>2.31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.29</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.4</td>\n",
       "      <td>58.0</td>\n",
       "      <td>334</td>\n",
       "      <td>4.20</td>\n",
       "      <td>4.23</td>\n",
       "      <td>2.63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.31</td>\n",
       "      <td>Good</td>\n",
       "      <td>J</td>\n",
       "      <td>SI2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>58.0</td>\n",
       "      <td>335</td>\n",
       "      <td>4.34</td>\n",
       "      <td>4.35</td>\n",
       "      <td>2.75</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat      cut color clarity  depth  table  price     x     y     z\n",
       "0   0.23    Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43\n",
       "1   0.21  Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31\n",
       "2   0.23     Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31\n",
       "3   0.29  Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63\n",
       "4   0.31     Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "diamonds.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the first 5 rows of the dataset using the `head` method. This by itself doesn't tell us much about the data that we have, but we can have a first look at the features (columns) and some of the values that each one takes.\n",
    "\n",
    "**What do you see? Make some comments about the values you see in each column, comparing them with the codebook. Is that what you would expect for these variables?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n",
    "'''\n",
    "There are 7 numeric columns (carat, depth, table, price, x, y, z) and 3 categorical (cut, color, clarity).\n",
    "Information visible in first 5 rows is in line with description provided in codebook.\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is very important to know the amount of data we have, because everything will depend on that, from the quality of the analysis to the choice of our infrastracture.\n",
    "\n",
    "**Check the shape of the data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(53940, 10)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "diamonds.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `clarity` column is confusing because we are not diamond experts. Let's create a new column with a new scale that is more understandable for us.\n",
    "\n",
    "**Create a new column with numbers from 0 to 7. The lowest would be 0 with value `I1` and the greatest 7 with value `IF`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['SI2', 'SI1', 'VS1', 'VS2', 'VVS2', 'VVS1', 'I1', 'IF'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds['clarity'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.5</td>\n",
       "      <td>55.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.95</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2.43</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>Premium</td>\n",
       "      <td>E</td>\n",
       "      <td>SI1</td>\n",
       "      <td>59.8</td>\n",
       "      <td>61.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.89</td>\n",
       "      <td>3.84</td>\n",
       "      <td>2.31</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Good</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>56.9</td>\n",
       "      <td>65.0</td>\n",
       "      <td>327</td>\n",
       "      <td>4.05</td>\n",
       "      <td>4.07</td>\n",
       "      <td>2.31</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.29</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.4</td>\n",
       "      <td>58.0</td>\n",
       "      <td>334</td>\n",
       "      <td>4.20</td>\n",
       "      <td>4.23</td>\n",
       "      <td>2.63</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.31</td>\n",
       "      <td>Good</td>\n",
       "      <td>J</td>\n",
       "      <td>SI2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>58.0</td>\n",
       "      <td>335</td>\n",
       "      <td>4.34</td>\n",
       "      <td>4.35</td>\n",
       "      <td>2.75</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.24</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>J</td>\n",
       "      <td>VVS2</td>\n",
       "      <td>62.8</td>\n",
       "      <td>57.0</td>\n",
       "      <td>336</td>\n",
       "      <td>3.94</td>\n",
       "      <td>3.96</td>\n",
       "      <td>2.48</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.24</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>I</td>\n",
       "      <td>VVS1</td>\n",
       "      <td>62.3</td>\n",
       "      <td>57.0</td>\n",
       "      <td>336</td>\n",
       "      <td>3.95</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2.47</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.26</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>H</td>\n",
       "      <td>SI1</td>\n",
       "      <td>61.9</td>\n",
       "      <td>55.0</td>\n",
       "      <td>337</td>\n",
       "      <td>4.07</td>\n",
       "      <td>4.11</td>\n",
       "      <td>2.53</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.22</td>\n",
       "      <td>Fair</td>\n",
       "      <td>E</td>\n",
       "      <td>VS2</td>\n",
       "      <td>65.1</td>\n",
       "      <td>61.0</td>\n",
       "      <td>337</td>\n",
       "      <td>3.87</td>\n",
       "      <td>3.78</td>\n",
       "      <td>2.49</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>H</td>\n",
       "      <td>VS1</td>\n",
       "      <td>59.4</td>\n",
       "      <td>61.0</td>\n",
       "      <td>338</td>\n",
       "      <td>4.00</td>\n",
       "      <td>4.05</td>\n",
       "      <td>2.39</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat        cut color clarity  depth  table  price     x     y     z  \\\n",
       "0   0.23      Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43   \n",
       "1   0.21    Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31   \n",
       "2   0.23       Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31   \n",
       "3   0.29    Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63   \n",
       "4   0.31       Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75   \n",
       "5   0.24  Very Good     J    VVS2   62.8   57.0    336  3.94  3.96  2.48   \n",
       "6   0.24  Very Good     I    VVS1   62.3   57.0    336  3.95  3.98  2.47   \n",
       "7   0.26  Very Good     H     SI1   61.9   55.0    337  4.07  4.11  2.53   \n",
       "8   0.22       Fair     E     VS2   65.1   61.0    337  3.87  3.78  2.49   \n",
       "9   0.23  Very Good     H     VS1   59.4   61.0    338  4.00  4.05  2.39   \n",
       "\n",
       "   clarity_rating  \n",
       "0               1  \n",
       "1               2  \n",
       "2               4  \n",
       "3               3  \n",
       "4               1  \n",
       "5               5  \n",
       "6               6  \n",
       "7               2  \n",
       "8               3  \n",
       "9               4  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "clarity_rating = {'I1':0, 'SI2':1, 'SI1':2, 'VS2':3, 'VS1':4, 'VVS2':5, 'VVS1':6, 'IF':7}\n",
    "diamonds['clarity_rating'] = [clarity_rating[i] for i in diamonds['clarity'] if i in clarity_rating.keys()]\n",
    "diamonds.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It makes sense to do the same with the `color` column.\n",
    "\n",
    "**Do the same with values from 0 to 6. Read the codebook to see the match**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.5</td>\n",
       "      <td>55.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.95</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2.43</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>Premium</td>\n",
       "      <td>E</td>\n",
       "      <td>SI1</td>\n",
       "      <td>59.8</td>\n",
       "      <td>61.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.89</td>\n",
       "      <td>3.84</td>\n",
       "      <td>2.31</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Good</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>56.9</td>\n",
       "      <td>65.0</td>\n",
       "      <td>327</td>\n",
       "      <td>4.05</td>\n",
       "      <td>4.07</td>\n",
       "      <td>2.31</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.29</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.4</td>\n",
       "      <td>58.0</td>\n",
       "      <td>334</td>\n",
       "      <td>4.20</td>\n",
       "      <td>4.23</td>\n",
       "      <td>2.63</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.31</td>\n",
       "      <td>Good</td>\n",
       "      <td>J</td>\n",
       "      <td>SI2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>58.0</td>\n",
       "      <td>335</td>\n",
       "      <td>4.34</td>\n",
       "      <td>4.35</td>\n",
       "      <td>2.75</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.24</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>J</td>\n",
       "      <td>VVS2</td>\n",
       "      <td>62.8</td>\n",
       "      <td>57.0</td>\n",
       "      <td>336</td>\n",
       "      <td>3.94</td>\n",
       "      <td>3.96</td>\n",
       "      <td>2.48</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.24</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>I</td>\n",
       "      <td>VVS1</td>\n",
       "      <td>62.3</td>\n",
       "      <td>57.0</td>\n",
       "      <td>336</td>\n",
       "      <td>3.95</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2.47</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.26</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>H</td>\n",
       "      <td>SI1</td>\n",
       "      <td>61.9</td>\n",
       "      <td>55.0</td>\n",
       "      <td>337</td>\n",
       "      <td>4.07</td>\n",
       "      <td>4.11</td>\n",
       "      <td>2.53</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.22</td>\n",
       "      <td>Fair</td>\n",
       "      <td>E</td>\n",
       "      <td>VS2</td>\n",
       "      <td>65.1</td>\n",
       "      <td>61.0</td>\n",
       "      <td>337</td>\n",
       "      <td>3.87</td>\n",
       "      <td>3.78</td>\n",
       "      <td>2.49</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>H</td>\n",
       "      <td>VS1</td>\n",
       "      <td>59.4</td>\n",
       "      <td>61.0</td>\n",
       "      <td>338</td>\n",
       "      <td>4.00</td>\n",
       "      <td>4.05</td>\n",
       "      <td>2.39</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat        cut color clarity  depth  table  price     x     y     z  \\\n",
       "0   0.23      Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43   \n",
       "1   0.21    Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31   \n",
       "2   0.23       Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31   \n",
       "3   0.29    Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63   \n",
       "4   0.31       Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75   \n",
       "5   0.24  Very Good     J    VVS2   62.8   57.0    336  3.94  3.96  2.48   \n",
       "6   0.24  Very Good     I    VVS1   62.3   57.0    336  3.95  3.98  2.47   \n",
       "7   0.26  Very Good     H     SI1   61.9   55.0    337  4.07  4.11  2.53   \n",
       "8   0.22       Fair     E     VS2   65.1   61.0    337  3.87  3.78  2.49   \n",
       "9   0.23  Very Good     H     VS1   59.4   61.0    338  4.00  4.05  2.39   \n",
       "\n",
       "   clarity_rating  color_rating  \n",
       "0               1             1  \n",
       "1               2             1  \n",
       "2               4             1  \n",
       "3               3             5  \n",
       "4               1             6  \n",
       "5               5             6  \n",
       "6               6             5  \n",
       "7               2             4  \n",
       "8               3             1  \n",
       "9               4             4  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "# diamonds['color'].unique()    #For some reason this syntax of 'unique' method doesn't sort values automatically.\n",
    "np.unique(diamonds['color'])\n",
    "color_rating = {'D':0, 'E':1, 'F':2, 'G':3, 'H':4, 'I':5, 'J':6}\n",
    "diamonds['color_rating'] = [color_rating[i] for i in diamonds['color'] if i in color_rating.keys()]\n",
    "diamonds.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the `info` method, we can see the features of the dataset, and the amount of observations (rows) that have a non-null value and the types of the features. \n",
    "\n",
    "**Now use the `info` method and comparing with the shape, comment on what you see**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(53940, 12)\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 53940 entries, 0 to 53939\n",
      "Data columns (total 12 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   carat           53940 non-null  float64\n",
      " 1   cut             53940 non-null  object \n",
      " 2   color           53940 non-null  object \n",
      " 3   clarity         53940 non-null  object \n",
      " 4   depth           53940 non-null  float64\n",
      " 5   table           53940 non-null  float64\n",
      " 6   price           53940 non-null  int64  \n",
      " 7   x               53940 non-null  float64\n",
      " 8   y               53940 non-null  float64\n",
      " 9   z               53940 non-null  float64\n",
      " 10  clarity_rating  53940 non-null  int64  \n",
      " 11  color_rating    53940 non-null  int64  \n",
      "dtypes: float64(6), int64(3), object(3)\n",
      "memory usage: 4.9+ MB\n",
      "None\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"\\nThere are no fields with Nan data. Usually it's a good sign although there are still might be errors within dataset.\\n\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "print(diamonds.shape)\n",
    "print(diamonds.info())\n",
    "'''\n",
    "There are no fields with Nan values. Usually it's a good sign although there are still might be errors within dataset.\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the last line of the info output, you have some information about the types of the columns. As you know, it is a good idea to check if the types of each column is what you expect. If a column has the right type, we will be able to do all the operations that we want to do. \n",
    "\n",
    "For instance, if we have a column that is a `date` with a `string` format, we will have the data but we won't be able to do a simple operation, such as format the date the way that we would like.\n",
    "\n",
    "Changing the data type to the one we needs can help us to solve a lot of problems in our data.\n",
    "\n",
    "**Check the types of each column and comment if it matches with the expected**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.5</td>\n",
       "      <td>55.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.95</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2.43</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>Premium</td>\n",
       "      <td>E</td>\n",
       "      <td>SI1</td>\n",
       "      <td>59.8</td>\n",
       "      <td>61.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.89</td>\n",
       "      <td>3.84</td>\n",
       "      <td>2.31</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Good</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>56.9</td>\n",
       "      <td>65.0</td>\n",
       "      <td>327</td>\n",
       "      <td>4.05</td>\n",
       "      <td>4.07</td>\n",
       "      <td>2.31</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.29</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.4</td>\n",
       "      <td>58.0</td>\n",
       "      <td>334</td>\n",
       "      <td>4.20</td>\n",
       "      <td>4.23</td>\n",
       "      <td>2.63</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.31</td>\n",
       "      <td>Good</td>\n",
       "      <td>J</td>\n",
       "      <td>SI2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>58.0</td>\n",
       "      <td>335</td>\n",
       "      <td>4.34</td>\n",
       "      <td>4.35</td>\n",
       "      <td>2.75</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat      cut color clarity  depth  table  price     x     y     z  \\\n",
       "0   0.23    Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43   \n",
       "1   0.21  Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31   \n",
       "2   0.23     Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31   \n",
       "3   0.29  Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63   \n",
       "4   0.31     Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75   \n",
       "\n",
       "   clarity_rating  color_rating  \n",
       "0               1             1  \n",
       "1               2             1  \n",
       "2               4             1  \n",
       "3               3             5  \n",
       "4               1             6  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n",
    "'''\n",
    "#   Column          Non-Null Count  Dtype  \n",
    "---  ------          --------------  -----  \n",
    " 0   carat           53940 non-null  float64    Numeric value and it usually fractional so float is suitable type.\n",
    " 1   cut             53940 non-null  object     Qualitative value and presented by English worlds, so object type is suitable.\n",
    " 2   color           53940 non-null  object     Qualitative value and presented by latin letters, so object type is suitable.\n",
    " 3   clarity         53940 non-null  object     Qualitative value and presented by code consisting of numbers and latin letters, object type is suitable.\n",
    " 4   depth           53940 non-null  float64    Numeric value and it usually fractional so float is suitable type.\n",
    " 5   table           53940 non-null  float64    Numeric value, currently presented by integers so maybe int type would be more suitable type, but it doesn´t have great impact on actions can be performed with data.\n",
    " 6   price           53940 non-null  int64      Numeric value, currently presented by integers but logically some prices can have cents. Anyway prices provided without cents so int is suitable type.\n",
    " 7   x               53940 non-null  float64    Numeric value and it usually fractional so float is suitable type.\n",
    " 8   y               53940 non-null  float64    Numeric value and it usually fractional so float is suitable type.\n",
    " 9   z               53940 non-null  float64    Numeric value and it usually fractional so float is suitable type.\n",
    " 10  clarity_rating  53940 non-null  int64      Qualitative value and presented by integers so int type is suitable.\n",
    " 11  color_rating    53940 non-null  int64      Qualitative value and presented by integers so int type is suitable.\n",
    "\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. A deeper look: checking the basic statistics.\n",
    "\n",
    "The `describe` method gives us an overview of our data. From here we can see all the descriptive metrics for our variables.\n",
    "\n",
    "**Use the `describe` method and comment on what you see**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.797940</td>\n",
       "      <td>61.749405</td>\n",
       "      <td>57.457184</td>\n",
       "      <td>3932.799722</td>\n",
       "      <td>5.731157</td>\n",
       "      <td>5.734526</td>\n",
       "      <td>3.538734</td>\n",
       "      <td>3.051020</td>\n",
       "      <td>2.594197</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.474011</td>\n",
       "      <td>1.432621</td>\n",
       "      <td>2.234491</td>\n",
       "      <td>3989.439738</td>\n",
       "      <td>1.121761</td>\n",
       "      <td>1.142135</td>\n",
       "      <td>0.705699</td>\n",
       "      <td>1.647136</td>\n",
       "      <td>1.701105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.200000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>326.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.400000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>56.000000</td>\n",
       "      <td>950.000000</td>\n",
       "      <td>4.710000</td>\n",
       "      <td>4.720000</td>\n",
       "      <td>2.910000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.700000</td>\n",
       "      <td>61.800000</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>2401.000000</td>\n",
       "      <td>5.700000</td>\n",
       "      <td>5.710000</td>\n",
       "      <td>3.530000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.040000</td>\n",
       "      <td>62.500000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>5324.250000</td>\n",
       "      <td>6.540000</td>\n",
       "      <td>6.540000</td>\n",
       "      <td>4.040000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>5.010000</td>\n",
       "      <td>79.000000</td>\n",
       "      <td>95.000000</td>\n",
       "      <td>18823.000000</td>\n",
       "      <td>10.740000</td>\n",
       "      <td>58.900000</td>\n",
       "      <td>31.800000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              carat         depth         table         price             x  \\\n",
       "count  53940.000000  53940.000000  53940.000000  53940.000000  53940.000000   \n",
       "mean       0.797940     61.749405     57.457184   3932.799722      5.731157   \n",
       "std        0.474011      1.432621      2.234491   3989.439738      1.121761   \n",
       "min        0.200000     43.000000     43.000000    326.000000      0.000000   \n",
       "25%        0.400000     61.000000     56.000000    950.000000      4.710000   \n",
       "50%        0.700000     61.800000     57.000000   2401.000000      5.700000   \n",
       "75%        1.040000     62.500000     59.000000   5324.250000      6.540000   \n",
       "max        5.010000     79.000000     95.000000  18823.000000     10.740000   \n",
       "\n",
       "                  y             z  clarity_rating  color_rating  \n",
       "count  53940.000000  53940.000000    53940.000000  53940.000000  \n",
       "mean       5.734526      3.538734        3.051020      2.594197  \n",
       "std        1.142135      0.705699        1.647136      1.701105  \n",
       "min        0.000000      0.000000        0.000000      0.000000  \n",
       "25%        4.720000      2.910000        2.000000      1.000000  \n",
       "50%        5.710000      3.530000        3.000000      3.000000  \n",
       "75%        6.540000      4.040000        4.000000      4.000000  \n",
       "max       58.900000     31.800000        7.000000      6.000000  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code and comments here\n",
    "\n",
    "'''\n",
    "There are zeros in columns x, y & z. Taking into account that these values represent dimentions of the stone it is deffinitelly\n",
    "an error and should be investigated further.\n",
    "Maximum values of y & z also look suspicious taking into account maximum value of 'table' and mean values for y & z respectively.\n",
    "They also should be investigated further.\n",
    "Remaining values look sensible from data quality point of view, more comments from analytical point of view will be provided\n",
    "in working file '2-EDA'.\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have probably noticed that the columns x, y and z have a minimum value of 0. This means that there are one or more rows (or observations) in our dataset that are supposedly representing a diamond that has lenght, width or depth of 0. Considering that we're talking about a physical object, this is impossible!\n",
    "\n",
    "Now let's proceed to check the rows that have a value of 0 in any of the x, y or z columns. By doing this we want to check if the data we are missing can be obtained using the data that we do have.\n",
    "\n",
    "**Check the columns with `x`, `y` and `z` with value 0 in all of them and comment what you see**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11963</th>\n",
       "      <td>1.00</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>H</td>\n",
       "      <td>VS2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>53.0</td>\n",
       "      <td>5139</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15951</th>\n",
       "      <td>1.14</td>\n",
       "      <td>Fair</td>\n",
       "      <td>G</td>\n",
       "      <td>VS1</td>\n",
       "      <td>57.5</td>\n",
       "      <td>67.0</td>\n",
       "      <td>6381</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24520</th>\n",
       "      <td>1.56</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>G</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.2</td>\n",
       "      <td>54.0</td>\n",
       "      <td>12800</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26243</th>\n",
       "      <td>1.20</td>\n",
       "      <td>Premium</td>\n",
       "      <td>D</td>\n",
       "      <td>VVS1</td>\n",
       "      <td>62.1</td>\n",
       "      <td>59.0</td>\n",
       "      <td>15686</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27429</th>\n",
       "      <td>2.25</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI2</td>\n",
       "      <td>62.8</td>\n",
       "      <td>59.0</td>\n",
       "      <td>18034</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49556</th>\n",
       "      <td>0.71</td>\n",
       "      <td>Good</td>\n",
       "      <td>F</td>\n",
       "      <td>SI2</td>\n",
       "      <td>64.1</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49557</th>\n",
       "      <td>0.71</td>\n",
       "      <td>Good</td>\n",
       "      <td>F</td>\n",
       "      <td>SI2</td>\n",
       "      <td>64.1</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat        cut color clarity  depth  table  price    x    y    z  \\\n",
       "11963   1.00  Very Good     H     VS2   63.3   53.0   5139  0.0  0.0  0.0   \n",
       "15951   1.14       Fair     G     VS1   57.5   67.0   6381  0.0  0.0  0.0   \n",
       "24520   1.56      Ideal     G     VS2   62.2   54.0  12800  0.0  0.0  0.0   \n",
       "26243   1.20    Premium     D    VVS1   62.1   59.0  15686  0.0  0.0  0.0   \n",
       "27429   2.25    Premium     H     SI2   62.8   59.0  18034  0.0  0.0  0.0   \n",
       "49556   0.71       Good     F     SI2   64.1   60.0   2130  0.0  0.0  0.0   \n",
       "49557   0.71       Good     F     SI2   64.1   60.0   2130  0.0  0.0  0.0   \n",
       "\n",
       "       clarity_rating  color_rating  \n",
       "11963               3             4  \n",
       "15951               4             3  \n",
       "24520               3             3  \n",
       "26243               6             0  \n",
       "27429               1             4  \n",
       "49556               1             2  \n",
       "49557               1             2  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "diamonds.loc[(diamonds['x'] == 0) & (diamonds['y'] == 0) & (diamonds['z'] == 0)]\n",
    "\n",
    "# There are 7 row where all three measures are missing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2207</th>\n",
       "      <td>1.00</td>\n",
       "      <td>Premium</td>\n",
       "      <td>G</td>\n",
       "      <td>SI2</td>\n",
       "      <td>59.1</td>\n",
       "      <td>59.0</td>\n",
       "      <td>3142</td>\n",
       "      <td>6.55</td>\n",
       "      <td>6.48</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2314</th>\n",
       "      <td>1.01</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>I1</td>\n",
       "      <td>58.1</td>\n",
       "      <td>59.0</td>\n",
       "      <td>3167</td>\n",
       "      <td>6.66</td>\n",
       "      <td>6.60</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4791</th>\n",
       "      <td>1.10</td>\n",
       "      <td>Premium</td>\n",
       "      <td>G</td>\n",
       "      <td>SI2</td>\n",
       "      <td>63.0</td>\n",
       "      <td>59.0</td>\n",
       "      <td>3696</td>\n",
       "      <td>6.50</td>\n",
       "      <td>6.47</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5471</th>\n",
       "      <td>1.01</td>\n",
       "      <td>Premium</td>\n",
       "      <td>F</td>\n",
       "      <td>SI2</td>\n",
       "      <td>59.2</td>\n",
       "      <td>58.0</td>\n",
       "      <td>3837</td>\n",
       "      <td>6.50</td>\n",
       "      <td>6.47</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10167</th>\n",
       "      <td>1.50</td>\n",
       "      <td>Good</td>\n",
       "      <td>G</td>\n",
       "      <td>I1</td>\n",
       "      <td>64.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>4731</td>\n",
       "      <td>7.15</td>\n",
       "      <td>7.04</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11182</th>\n",
       "      <td>1.07</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>F</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.6</td>\n",
       "      <td>56.0</td>\n",
       "      <td>4954</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.62</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11963</th>\n",
       "      <td>1.00</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>H</td>\n",
       "      <td>VS2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>53.0</td>\n",
       "      <td>5139</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13601</th>\n",
       "      <td>1.15</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>G</td>\n",
       "      <td>VS2</td>\n",
       "      <td>59.2</td>\n",
       "      <td>56.0</td>\n",
       "      <td>5564</td>\n",
       "      <td>6.88</td>\n",
       "      <td>6.83</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15951</th>\n",
       "      <td>1.14</td>\n",
       "      <td>Fair</td>\n",
       "      <td>G</td>\n",
       "      <td>VS1</td>\n",
       "      <td>57.5</td>\n",
       "      <td>67.0</td>\n",
       "      <td>6381</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24394</th>\n",
       "      <td>2.18</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI2</td>\n",
       "      <td>59.4</td>\n",
       "      <td>61.0</td>\n",
       "      <td>12631</td>\n",
       "      <td>8.49</td>\n",
       "      <td>8.45</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24520</th>\n",
       "      <td>1.56</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>G</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.2</td>\n",
       "      <td>54.0</td>\n",
       "      <td>12800</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26123</th>\n",
       "      <td>2.25</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>SI1</td>\n",
       "      <td>61.3</td>\n",
       "      <td>58.0</td>\n",
       "      <td>15397</td>\n",
       "      <td>8.52</td>\n",
       "      <td>8.42</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26243</th>\n",
       "      <td>1.20</td>\n",
       "      <td>Premium</td>\n",
       "      <td>D</td>\n",
       "      <td>VVS1</td>\n",
       "      <td>62.1</td>\n",
       "      <td>59.0</td>\n",
       "      <td>15686</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27112</th>\n",
       "      <td>2.20</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI1</td>\n",
       "      <td>61.2</td>\n",
       "      <td>59.0</td>\n",
       "      <td>17265</td>\n",
       "      <td>8.42</td>\n",
       "      <td>8.37</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27429</th>\n",
       "      <td>2.25</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI2</td>\n",
       "      <td>62.8</td>\n",
       "      <td>59.0</td>\n",
       "      <td>18034</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27503</th>\n",
       "      <td>2.02</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.7</td>\n",
       "      <td>53.0</td>\n",
       "      <td>18207</td>\n",
       "      <td>8.02</td>\n",
       "      <td>7.95</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27739</th>\n",
       "      <td>2.80</td>\n",
       "      <td>Good</td>\n",
       "      <td>G</td>\n",
       "      <td>SI2</td>\n",
       "      <td>63.8</td>\n",
       "      <td>58.0</td>\n",
       "      <td>18788</td>\n",
       "      <td>8.90</td>\n",
       "      <td>8.85</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49556</th>\n",
       "      <td>0.71</td>\n",
       "      <td>Good</td>\n",
       "      <td>F</td>\n",
       "      <td>SI2</td>\n",
       "      <td>64.1</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49557</th>\n",
       "      <td>0.71</td>\n",
       "      <td>Good</td>\n",
       "      <td>F</td>\n",
       "      <td>SI2</td>\n",
       "      <td>64.1</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51506</th>\n",
       "      <td>1.12</td>\n",
       "      <td>Premium</td>\n",
       "      <td>G</td>\n",
       "      <td>I1</td>\n",
       "      <td>60.4</td>\n",
       "      <td>59.0</td>\n",
       "      <td>2383</td>\n",
       "      <td>6.71</td>\n",
       "      <td>6.67</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat        cut color clarity  depth  table  price     x     y    z  \\\n",
       "2207    1.00    Premium     G     SI2   59.1   59.0   3142  6.55  6.48  0.0   \n",
       "2314    1.01    Premium     H      I1   58.1   59.0   3167  6.66  6.60  0.0   \n",
       "4791    1.10    Premium     G     SI2   63.0   59.0   3696  6.50  6.47  0.0   \n",
       "5471    1.01    Premium     F     SI2   59.2   58.0   3837  6.50  6.47  0.0   \n",
       "10167   1.50       Good     G      I1   64.0   61.0   4731  7.15  7.04  0.0   \n",
       "11182   1.07      Ideal     F     SI2   61.6   56.0   4954  0.00  6.62  0.0   \n",
       "11963   1.00  Very Good     H     VS2   63.3   53.0   5139  0.00  0.00  0.0   \n",
       "13601   1.15      Ideal     G     VS2   59.2   56.0   5564  6.88  6.83  0.0   \n",
       "15951   1.14       Fair     G     VS1   57.5   67.0   6381  0.00  0.00  0.0   \n",
       "24394   2.18    Premium     H     SI2   59.4   61.0  12631  8.49  8.45  0.0   \n",
       "24520   1.56      Ideal     G     VS2   62.2   54.0  12800  0.00  0.00  0.0   \n",
       "26123   2.25    Premium     I     SI1   61.3   58.0  15397  8.52  8.42  0.0   \n",
       "26243   1.20    Premium     D    VVS1   62.1   59.0  15686  0.00  0.00  0.0   \n",
       "27112   2.20    Premium     H     SI1   61.2   59.0  17265  8.42  8.37  0.0   \n",
       "27429   2.25    Premium     H     SI2   62.8   59.0  18034  0.00  0.00  0.0   \n",
       "27503   2.02    Premium     H     VS2   62.7   53.0  18207  8.02  7.95  0.0   \n",
       "27739   2.80       Good     G     SI2   63.8   58.0  18788  8.90  8.85  0.0   \n",
       "49556   0.71       Good     F     SI2   64.1   60.0   2130  0.00  0.00  0.0   \n",
       "49557   0.71       Good     F     SI2   64.1   60.0   2130  0.00  0.00  0.0   \n",
       "51506   1.12    Premium     G      I1   60.4   59.0   2383  6.71  6.67  0.0   \n",
       "\n",
       "       clarity_rating  color_rating  \n",
       "2207                1             3  \n",
       "2314                0             4  \n",
       "4791                1             3  \n",
       "5471                1             2  \n",
       "10167               0             3  \n",
       "11182               1             2  \n",
       "11963               3             4  \n",
       "13601               3             3  \n",
       "15951               4             3  \n",
       "24394               1             4  \n",
       "24520               3             3  \n",
       "26123               2             5  \n",
       "26243               6             0  \n",
       "27112               2             4  \n",
       "27429               1             4  \n",
       "27503               3             4  \n",
       "27739               1             3  \n",
       "49556               1             2  \n",
       "49557               1             2  \n",
       "51506               0             3  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.loc[(diamonds['x'] == 0) | (diamonds['y'] == 0)| (diamonds['z'] == 0)]\n",
    "\n",
    "# There are 20 rows where eigher of three measures or all of them are missing. Majority of them have x & y values but z is missing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we have 20 rows that have a value of 0 in some or all the aforementioned columns.\n",
    "Most of them (12) are missing the z value, which we can obtain using the columns depth, x and y. \n",
    "\n",
    "20 rows with issues represent just 0.03% of our data (20 out of 53940) so it wouldn't be a big deal to remove them. Still, lets try to keep all the data we have. \n",
    "\n",
    "For those 12 rows, we will create a function that applies the formula given in the codebook and get the value of z. We will drop the other rows (8), since they are missing all 3 values or 2 of them.\n",
    "\n",
    "**Create a function named `calculate_z` that applies the function in the codebook to one single row you give to the function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n",
    "\n",
    "# Total depth percentage = z / mean(x, y) = 2 * z / (x + y)\n",
    "# hense z = (d*(x+y))/2\n",
    "def calculate_z(row):\n",
    "    return round((row['depth']/100*(row['x']+row['y']))/2, 2)\n",
    "    \n",
    "# calculate_z(diamonds.iloc[2207])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Apply it just to the rows with incorrect values**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "diamonds['z'] = np.where(diamonds['z'] == 0, diamonds.apply(lambda row: calculate_z(row), axis=1), diamonds['z'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "carat                   1\n",
       "cut               Premium\n",
       "color                   G\n",
       "clarity               SI2\n",
       "depth                59.1\n",
       "table                  59\n",
       "price                3142\n",
       "x                    6.55\n",
       "y                    6.48\n",
       "z                    3.85\n",
       "clarity_rating          1\n",
       "color_rating            3\n",
       "Name: 2207, dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check one known row is updated.\n",
    "diamonds.iloc[2207]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11182</th>\n",
       "      <td>1.07</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>F</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.6</td>\n",
       "      <td>56.0</td>\n",
       "      <td>4954</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.62</td>\n",
       "      <td>2.04</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11963</th>\n",
       "      <td>1.00</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>H</td>\n",
       "      <td>VS2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>53.0</td>\n",
       "      <td>5139</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15951</th>\n",
       "      <td>1.14</td>\n",
       "      <td>Fair</td>\n",
       "      <td>G</td>\n",
       "      <td>VS1</td>\n",
       "      <td>57.5</td>\n",
       "      <td>67.0</td>\n",
       "      <td>6381</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24520</th>\n",
       "      <td>1.56</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>G</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.2</td>\n",
       "      <td>54.0</td>\n",
       "      <td>12800</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26243</th>\n",
       "      <td>1.20</td>\n",
       "      <td>Premium</td>\n",
       "      <td>D</td>\n",
       "      <td>VVS1</td>\n",
       "      <td>62.1</td>\n",
       "      <td>59.0</td>\n",
       "      <td>15686</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27429</th>\n",
       "      <td>2.25</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI2</td>\n",
       "      <td>62.8</td>\n",
       "      <td>59.0</td>\n",
       "      <td>18034</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49556</th>\n",
       "      <td>0.71</td>\n",
       "      <td>Good</td>\n",
       "      <td>F</td>\n",
       "      <td>SI2</td>\n",
       "      <td>64.1</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49557</th>\n",
       "      <td>0.71</td>\n",
       "      <td>Good</td>\n",
       "      <td>F</td>\n",
       "      <td>SI2</td>\n",
       "      <td>64.1</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2130</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat        cut color clarity  depth  table  price    x     y     z  \\\n",
       "11182   1.07      Ideal     F     SI2   61.6   56.0   4954  0.0  6.62  2.04   \n",
       "11963   1.00  Very Good     H     VS2   63.3   53.0   5139  0.0  0.00  0.00   \n",
       "15951   1.14       Fair     G     VS1   57.5   67.0   6381  0.0  0.00  0.00   \n",
       "24520   1.56      Ideal     G     VS2   62.2   54.0  12800  0.0  0.00  0.00   \n",
       "26243   1.20    Premium     D    VVS1   62.1   59.0  15686  0.0  0.00  0.00   \n",
       "27429   2.25    Premium     H     SI2   62.8   59.0  18034  0.0  0.00  0.00   \n",
       "49556   0.71       Good     F     SI2   64.1   60.0   2130  0.0  0.00  0.00   \n",
       "49557   0.71       Good     F     SI2   64.1   60.0   2130  0.0  0.00  0.00   \n",
       "\n",
       "       clarity_rating  color_rating  \n",
       "11182               1             2  \n",
       "11963               3             4  \n",
       "15951               4             3  \n",
       "24520               3             3  \n",
       "26243               6             0  \n",
       "27429               1             4  \n",
       "49556               1             2  \n",
       "49557               1             2  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary check to confirm there are no zeros in z anymore.\n",
    "diamonds.loc[(diamonds['x'] == 0) | (diamonds['y'] == 0)| (diamonds['z'] == 0)]\n",
    "# There are still 8 rows with missing values in x or/and y so z can't be calculated as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we leave the other 8 values as they are, it would negatively affect our analysis, because these are data that do not make logical sense. Therefore it is better to consider those values as NaN values, since they are probably the result of a mistake or error during process of measuring and storing these values in a dataset.\n",
    "\n",
    "To replace them we can use the pandas .replace() method and np.NaN.\n",
    "\n",
    "**Replace the zero values in the `z` column for a NaN**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "carat                     1\n",
       "cut               Very Good\n",
       "color                     H\n",
       "clarity                 VS2\n",
       "depth                  63.3\n",
       "table                    53\n",
       "price                  5139\n",
       "x                       NaN\n",
       "y                       NaN\n",
       "z                       NaN\n",
       "clarity_rating            3\n",
       "color_rating              4\n",
       "Name: 11963, dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "# It actually needs to be done for x, y & z and not only for z if we want to be able to see correct min values before dropping\n",
    "# any rows as it's requested in section below.\n",
    "\n",
    "# d['z'].replace(0, np.NaN) # Why doesn't work?\n",
    "diamonds['z'] = np.where(diamonds['z'] == 0, np.NaN, diamonds['z'])\n",
    "diamonds['x'] = np.where(diamonds['x'] == 0, np.NaN, diamonds['x'])\n",
    "diamonds['y'] = np.where(diamonds['y'] == 0, np.NaN, diamonds['y'])\n",
    "\n",
    "# Check known row that it worked.\n",
    "diamonds.iloc[11963]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "# Bonus: check the new z values\n",
    "Since we need to be 100% sure of our data, let's create a function that validates our z. To do so, we will use the same formula, but this time we will calculate the value of depth with the new value assigned to z.\n",
    "\n",
    "**Create a function named `validate_z` that compares the `z`  in cells above with the one thrown by the formula and run it with the rows you changed in the cells above**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#your code here\n",
    "# Total depth percentage = z / mean(x, y) = 2 * z / (x + y)\n",
    "def validate_z(row):\n",
    "    d_calc = round(2*row['z']/(row['x']+row['y'])*100, 1)    # Calculate depth % based on x, y, z\n",
    "    return round((d_calc - row['depth']), 1)                 # Compare calculated value with given value\n",
    "    \n",
    "# validate_z(diamonds.iloc[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# I would like to check all my depth values and not only in rows where I changed z:\n",
    "diamonds['depth_variance'] = diamonds.apply(lambda row: validate_z(row), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "carat              1.15\n",
       "cut               Ideal\n",
       "color                 G\n",
       "clarity             VS2\n",
       "depth              59.2\n",
       "table                56\n",
       "price              5564\n",
       "x                  6.88\n",
       "y                  6.83\n",
       "z                  4.06\n",
       "clarity_rating        3\n",
       "color_rating          3\n",
       "depth_variance        0\n",
       "Name: 13601, dtype: object"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.iloc[13601]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "      <th>depth_variance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2274</th>\n",
       "      <td>1.22</td>\n",
       "      <td>Premium</td>\n",
       "      <td>J</td>\n",
       "      <td>SI2</td>\n",
       "      <td>62.6</td>\n",
       "      <td>59.0</td>\n",
       "      <td>3156</td>\n",
       "      <td>6.79</td>\n",
       "      <td>4.24</td>\n",
       "      <td>3.76</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>5.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2349</th>\n",
       "      <td>0.70</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>G</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.7</td>\n",
       "      <td>54.0</td>\n",
       "      <td>3172</td>\n",
       "      <td>5.65</td>\n",
       "      <td>5.70</td>\n",
       "      <td>3.65</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2941</th>\n",
       "      <td>1.00</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>J</td>\n",
       "      <td>SI2</td>\n",
       "      <td>62.8</td>\n",
       "      <td>63.0</td>\n",
       "      <td>3293</td>\n",
       "      <td>6.26</td>\n",
       "      <td>6.19</td>\n",
       "      <td>3.19</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>-11.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3511</th>\n",
       "      <td>0.70</td>\n",
       "      <td>Premium</td>\n",
       "      <td>E</td>\n",
       "      <td>IF</td>\n",
       "      <td>62.9</td>\n",
       "      <td>59.0</td>\n",
       "      <td>3403</td>\n",
       "      <td>5.66</td>\n",
       "      <td>5.59</td>\n",
       "      <td>3.40</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>-2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4124</th>\n",
       "      <td>1.01</td>\n",
       "      <td>Fair</td>\n",
       "      <td>F</td>\n",
       "      <td>SI2</td>\n",
       "      <td>64.6</td>\n",
       "      <td>59.0</td>\n",
       "      <td>3540</td>\n",
       "      <td>6.19</td>\n",
       "      <td>6.25</td>\n",
       "      <td>4.20</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51860</th>\n",
       "      <td>0.76</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI1</td>\n",
       "      <td>60.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2425</td>\n",
       "      <td>5.80</td>\n",
       "      <td>5.76</td>\n",
       "      <td>3.76</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>5.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52260</th>\n",
       "      <td>0.78</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI1</td>\n",
       "      <td>60.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2489</td>\n",
       "      <td>5.93</td>\n",
       "      <td>5.92</td>\n",
       "      <td>3.62</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53004</th>\n",
       "      <td>0.70</td>\n",
       "      <td>Premium</td>\n",
       "      <td>F</td>\n",
       "      <td>SI1</td>\n",
       "      <td>60.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2596</td>\n",
       "      <td>5.62</td>\n",
       "      <td>5.55</td>\n",
       "      <td>3.60</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>4.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53005</th>\n",
       "      <td>0.70</td>\n",
       "      <td>Premium</td>\n",
       "      <td>D</td>\n",
       "      <td>SI2</td>\n",
       "      <td>60.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2596</td>\n",
       "      <td>5.72</td>\n",
       "      <td>5.56</td>\n",
       "      <td>3.50</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53727</th>\n",
       "      <td>0.78</td>\n",
       "      <td>Fair</td>\n",
       "      <td>E</td>\n",
       "      <td>SI2</td>\n",
       "      <td>66.9</td>\n",
       "      <td>57.0</td>\n",
       "      <td>2721</td>\n",
       "      <td>5.70</td>\n",
       "      <td>5.66</td>\n",
       "      <td>3.60</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-3.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>72 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat        cut color clarity  depth  table  price     x     y     z  \\\n",
       "2274    1.22    Premium     J     SI2   62.6   59.0   3156  6.79  4.24  3.76   \n",
       "2349    0.70      Ideal     G     VS2   62.7   54.0   3172  5.65  5.70  3.65   \n",
       "2941    1.00  Very Good     J     SI2   62.8   63.0   3293  6.26  6.19  3.19   \n",
       "3511    0.70    Premium     E      IF   62.9   59.0   3403  5.66  5.59  3.40   \n",
       "4124    1.01       Fair     F     SI2   64.6   59.0   3540  6.19  6.25  4.20   \n",
       "...      ...        ...   ...     ...    ...    ...    ...   ...   ...   ...   \n",
       "51860   0.76    Premium     H     SI1   60.0   60.0   2425  5.80  5.76  3.76   \n",
       "52260   0.78    Premium     H     SI1   60.0   60.0   2489  5.93  5.92  3.62   \n",
       "53004   0.70    Premium     F     SI1   60.0   60.0   2596  5.62  5.55  3.60   \n",
       "53005   0.70    Premium     D     SI2   60.0   60.0   2596  5.72  5.56  3.50   \n",
       "53727   0.78       Fair     E     SI2   66.9   57.0   2721  5.70  5.66  3.60   \n",
       "\n",
       "       clarity_rating  color_rating  depth_variance  \n",
       "2274                1             6             5.6  \n",
       "2349                3             3             1.6  \n",
       "2941                1             6           -11.6  \n",
       "3511                7             1            -2.5  \n",
       "4124                1             2             2.9  \n",
       "...               ...           ...             ...  \n",
       "51860               2             4             5.1  \n",
       "52260               2             4             1.1  \n",
       "53004               2             2             4.5  \n",
       "53005               1             0             2.1  \n",
       "53727               1             1            -3.5  \n",
       "\n",
       "[72 rows x 13 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's say variance below 1% is irrelevant and how many relevant variances we have.\n",
    "check = diamonds.loc[(diamonds['depth_variance'] > 1) | (diamonds['depth_variance'] < -1)]\n",
    "check\n",
    "# It's actually quite a lot, 72 values or 0.13% of the population."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>depth_variance</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"10\" valign=\"top\">Fair</th>\n",
       "      <th>D</th>\n",
       "      <th>VS1</th>\n",
       "      <td>9.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>E</th>\n",
       "      <th>SI2</th>\n",
       "      <td>-3.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>F</th>\n",
       "      <th>SI2</th>\n",
       "      <td>2.900000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">G</th>\n",
       "      <th>SI1</th>\n",
       "      <td>2.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SI2</th>\n",
       "      <td>12.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS2</th>\n",
       "      <td>21.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">H</th>\n",
       "      <th>VS1</th>\n",
       "      <td>-10.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS2</th>\n",
       "      <td>2.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">I</th>\n",
       "      <th>I1</th>\n",
       "      <td>2.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS1</th>\n",
       "      <td>1.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Good</th>\n",
       "      <th>E</th>\n",
       "      <th>VS2</th>\n",
       "      <td>-10.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"15\" valign=\"top\">Ideal</th>\n",
       "      <th>D</th>\n",
       "      <th>SI1</th>\n",
       "      <td>-1.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">E</th>\n",
       "      <th>VS1</th>\n",
       "      <td>-16.450000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VVS1</th>\n",
       "      <td>-6.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VVS2</th>\n",
       "      <td>-17.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">F</th>\n",
       "      <th>SI1</th>\n",
       "      <td>-44.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS1</th>\n",
       "      <td>1.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VVS1</th>\n",
       "      <td>2.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VVS2</th>\n",
       "      <td>-5.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">G</th>\n",
       "      <th>SI1</th>\n",
       "      <td>-2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS2</th>\n",
       "      <td>1.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VVS1</th>\n",
       "      <td>-1.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">H</th>\n",
       "      <th>SI2</th>\n",
       "      <td>4.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS1</th>\n",
       "      <td>-41.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I</th>\n",
       "      <th>SI1</th>\n",
       "      <td>-41.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>J</th>\n",
       "      <th>VS2</th>\n",
       "      <td>20.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"18\" valign=\"top\">Premium</th>\n",
       "      <th>D</th>\n",
       "      <th>SI2</th>\n",
       "      <td>2.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">E</th>\n",
       "      <th>IF</th>\n",
       "      <td>-2.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SI1</th>\n",
       "      <td>-1.950000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS1</th>\n",
       "      <td>-1.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS2</th>\n",
       "      <td>3.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VVS2</th>\n",
       "      <td>1.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">F</th>\n",
       "      <th>SI1</th>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SI2</th>\n",
       "      <td>0.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS1</th>\n",
       "      <td>-4.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">G</th>\n",
       "      <th>SI1</th>\n",
       "      <td>5.266667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SI2</th>\n",
       "      <td>-0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS2</th>\n",
       "      <td>-1.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"4\" valign=\"top\">H</th>\n",
       "      <th>SI1</th>\n",
       "      <td>1.220000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SI2</th>\n",
       "      <td>1.950000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS1</th>\n",
       "      <td>7.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS2</th>\n",
       "      <td>0.933333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">J</th>\n",
       "      <th>SI2</th>\n",
       "      <td>5.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS1</th>\n",
       "      <td>2.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">Very Good</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">E</th>\n",
       "      <th>VS1</th>\n",
       "      <td>557.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VS2</th>\n",
       "      <td>-18.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">G</th>\n",
       "      <th>SI1</th>\n",
       "      <td>-2.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VVS1</th>\n",
       "      <td>37.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>J</th>\n",
       "      <th>SI2</th>\n",
       "      <td>-11.600000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         depth_variance\n",
       "cut       color clarity                \n",
       "Fair      D     VS1            9.100000\n",
       "          E     SI2           -3.500000\n",
       "          F     SI2            2.900000\n",
       "          G     SI1            2.550000\n",
       "                SI2           12.700000\n",
       "                VS2           21.700000\n",
       "          H     VS1          -10.400000\n",
       "                VS2            2.300000\n",
       "          I     I1             2.600000\n",
       "                VS1            1.700000\n",
       "Good      E     VS2          -10.100000\n",
       "Ideal     D     SI1           -1.100000\n",
       "          E     VS1          -16.450000\n",
       "                VVS1          -6.500000\n",
       "                VVS2         -17.000000\n",
       "          F     SI1          -44.500000\n",
       "                VS1            1.800000\n",
       "                VVS1           2.600000\n",
       "                VVS2          -5.500000\n",
       "          G     SI1           -2.000000\n",
       "                VS2            1.600000\n",
       "                VVS1          -1.600000\n",
       "          H     SI2            4.700000\n",
       "                VS1          -41.300000\n",
       "          I     SI1          -41.400000\n",
       "          J     VS2           20.000000\n",
       "Premium   D     SI2            2.100000\n",
       "          E     IF            -2.500000\n",
       "                SI1           -1.950000\n",
       "                VS1           -1.100000\n",
       "                VS2            3.800000\n",
       "                VVS2           1.400000\n",
       "          F     SI1            0.500000\n",
       "                SI2            0.300000\n",
       "                VS1           -4.200000\n",
       "          G     SI1            5.266667\n",
       "                SI2           -0.500000\n",
       "                VS2           -1.700000\n",
       "          H     SI1            1.220000\n",
       "                SI2            1.950000\n",
       "                VS1            7.200000\n",
       "                VS2            0.933333\n",
       "          J     SI2            5.600000\n",
       "                VS1            2.100000\n",
       "Very Good E     VS1          557.500000\n",
       "                VS2          -18.700000\n",
       "          G     SI1           -2.200000\n",
       "                VVS1          37.000000\n",
       "          J     SI2          -11.600000"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "check.pivot_table(index=['cut', 'color', 'clarity'], values=['depth_variance'])\n",
    "# For some categories average variance is above 10%.\n",
    "# Assuming that x, y & z are correct values I suggest to add new column 'depth_calculated' and compare main statistical measurements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_depth(row):\n",
    "    return round(2*row['z']/(row['x']+row['y'])*100, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "      <th>depth_variance</th>\n",
       "      <th>depth_calc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.5</td>\n",
       "      <td>55.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.95</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2.43</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-0.2</td>\n",
       "      <td>61.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>Premium</td>\n",
       "      <td>E</td>\n",
       "      <td>SI1</td>\n",
       "      <td>59.8</td>\n",
       "      <td>61.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.89</td>\n",
       "      <td>3.84</td>\n",
       "      <td>2.31</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>59.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Good</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>56.9</td>\n",
       "      <td>65.0</td>\n",
       "      <td>327</td>\n",
       "      <td>4.05</td>\n",
       "      <td>4.07</td>\n",
       "      <td>2.31</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>56.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.29</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.4</td>\n",
       "      <td>58.0</td>\n",
       "      <td>334</td>\n",
       "      <td>4.20</td>\n",
       "      <td>4.23</td>\n",
       "      <td>2.63</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>62.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.31</td>\n",
       "      <td>Good</td>\n",
       "      <td>J</td>\n",
       "      <td>SI2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>58.0</td>\n",
       "      <td>335</td>\n",
       "      <td>4.34</td>\n",
       "      <td>4.35</td>\n",
       "      <td>2.75</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>63.3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat      cut color clarity  depth  table  price     x     y     z  \\\n",
       "0   0.23    Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43   \n",
       "1   0.21  Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31   \n",
       "2   0.23     Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31   \n",
       "3   0.29  Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63   \n",
       "4   0.31     Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75   \n",
       "\n",
       "   clarity_rating  color_rating  depth_variance  depth_calc  \n",
       "0               1             1            -0.2        61.3  \n",
       "1               2             1             0.0        59.8  \n",
       "2               4             1             0.0        56.9  \n",
       "3               3             5             0.0        62.4  \n",
       "4               1             6             0.0        63.3  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds['depth_calc'] = diamonds.apply(lambda row: calculate_depth(row), axis=1)\n",
    "diamonds.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "      <th>depth_variance</th>\n",
       "      <th>depth_calc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53933.000000</td>\n",
       "      <td>53933.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53940.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.797940</td>\n",
       "      <td>61.749405</td>\n",
       "      <td>57.457184</td>\n",
       "      <td>3932.799722</td>\n",
       "      <td>5.732007</td>\n",
       "      <td>5.735270</td>\n",
       "      <td>3.540238</td>\n",
       "      <td>3.051020</td>\n",
       "      <td>2.594197</td>\n",
       "      <td>0.009243</td>\n",
       "      <td>61.758579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.474011</td>\n",
       "      <td>1.432621</td>\n",
       "      <td>2.234491</td>\n",
       "      <td>3989.439738</td>\n",
       "      <td>1.119670</td>\n",
       "      <td>1.140339</td>\n",
       "      <td>0.702690</td>\n",
       "      <td>1.647136</td>\n",
       "      <td>1.701105</td>\n",
       "      <td>2.452531</td>\n",
       "      <td>2.833443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.200000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>326.000000</td>\n",
       "      <td>3.730000</td>\n",
       "      <td>3.680000</td>\n",
       "      <td>1.070000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-44.500000</td>\n",
       "      <td>16.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.400000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>56.000000</td>\n",
       "      <td>950.000000</td>\n",
       "      <td>4.710000</td>\n",
       "      <td>4.720000</td>\n",
       "      <td>2.910000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.700000</td>\n",
       "      <td>61.800000</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>2401.000000</td>\n",
       "      <td>5.700000</td>\n",
       "      <td>5.710000</td>\n",
       "      <td>3.530000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>61.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.040000</td>\n",
       "      <td>62.500000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>5324.250000</td>\n",
       "      <td>6.540000</td>\n",
       "      <td>6.540000</td>\n",
       "      <td>4.040000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>62.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>5.010000</td>\n",
       "      <td>79.000000</td>\n",
       "      <td>95.000000</td>\n",
       "      <td>18823.000000</td>\n",
       "      <td>10.740000</td>\n",
       "      <td>58.900000</td>\n",
       "      <td>31.800000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>557.500000</td>\n",
       "      <td>619.300000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              carat         depth         table         price             x  \\\n",
       "count  53940.000000  53940.000000  53940.000000  53940.000000  53932.000000   \n",
       "mean       0.797940     61.749405     57.457184   3932.799722      5.732007   \n",
       "std        0.474011      1.432621      2.234491   3989.439738      1.119670   \n",
       "min        0.200000     43.000000     43.000000    326.000000      3.730000   \n",
       "25%        0.400000     61.000000     56.000000    950.000000      4.710000   \n",
       "50%        0.700000     61.800000     57.000000   2401.000000      5.700000   \n",
       "75%        1.040000     62.500000     59.000000   5324.250000      6.540000   \n",
       "max        5.010000     79.000000     95.000000  18823.000000     10.740000   \n",
       "\n",
       "                  y             z  clarity_rating  color_rating  \\\n",
       "count  53933.000000  53933.000000    53940.000000  53940.000000   \n",
       "mean       5.735270      3.540238        3.051020      2.594197   \n",
       "std        1.140339      0.702690        1.647136      1.701105   \n",
       "min        3.680000      1.070000        0.000000      0.000000   \n",
       "25%        4.720000      2.910000        2.000000      1.000000   \n",
       "50%        5.710000      3.530000        3.000000      3.000000   \n",
       "75%        6.540000      4.040000        4.000000      4.000000   \n",
       "max       58.900000     31.800000        7.000000      6.000000   \n",
       "\n",
       "       depth_variance    depth_calc  \n",
       "count    53932.000000  53932.000000  \n",
       "mean         0.009243     61.758579  \n",
       "std          2.452531      2.833443  \n",
       "min        -44.500000     16.100000  \n",
       "25%          0.000000     61.000000  \n",
       "50%          0.000000     61.800000  \n",
       "75%          0.000000     62.500000  \n",
       "max        557.500000    619.300000  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "As per summary above it seems it's not necessary to rewrite depth as major changes are in min and max and they don't seem to\n",
    "be correct.\n",
    "I will drop two columns I've added eearlier for check as they take space and in real world with real data it might matter.\n",
    "Also it can be confusing for other users of data.\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.5</td>\n",
       "      <td>55.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.95</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2.43</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>Premium</td>\n",
       "      <td>E</td>\n",
       "      <td>SI1</td>\n",
       "      <td>59.8</td>\n",
       "      <td>61.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.89</td>\n",
       "      <td>3.84</td>\n",
       "      <td>2.31</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Good</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>56.9</td>\n",
       "      <td>65.0</td>\n",
       "      <td>327</td>\n",
       "      <td>4.05</td>\n",
       "      <td>4.07</td>\n",
       "      <td>2.31</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.29</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.4</td>\n",
       "      <td>58.0</td>\n",
       "      <td>334</td>\n",
       "      <td>4.20</td>\n",
       "      <td>4.23</td>\n",
       "      <td>2.63</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.31</td>\n",
       "      <td>Good</td>\n",
       "      <td>J</td>\n",
       "      <td>SI2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>58.0</td>\n",
       "      <td>335</td>\n",
       "      <td>4.34</td>\n",
       "      <td>4.35</td>\n",
       "      <td>2.75</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   carat      cut color clarity  depth  table  price     x     y     z  \\\n",
       "0   0.23    Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43   \n",
       "1   0.21  Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31   \n",
       "2   0.23     Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31   \n",
       "3   0.29  Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63   \n",
       "4   0.31     Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75   \n",
       "\n",
       "   clarity_rating  color_rating  \n",
       "0               1             1  \n",
       "1               2             1  \n",
       "2               4             1  \n",
       "3               3             5  \n",
       "4               1             6  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.drop(['depth_variance', 'depth_calc'], axis=1, inplace=True)\n",
    "diamonds.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the data again with the `describe()` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The minimum value for x, y and z should now be a positive number, as it should be for the physical measurements of an object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's finish by checking for NaN values in the data. Since we introduced them ourselves using 'replace', we will surely find some, but there may be more that are unrelated to the x, y and z columns. Checking NaNs is a fundamental part of data cleaning and it's always better to do this kind of operations before proceeding with analysis.\n",
    "\n",
    "**Check how many NaNs do you have, comment what you would do with those values, and then do so**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "carat             0\n",
       "cut               0\n",
       "color             0\n",
       "clarity           0\n",
       "depth             0\n",
       "table             0\n",
       "price             0\n",
       "x                 8\n",
       "y                 7\n",
       "z                 7\n",
       "clarity_rating    0\n",
       "color_rating      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "diamonds.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.014831294030404153\n"
     ]
    }
   ],
   "source": [
    "print(8/53940*100)\n",
    "#Number of observations without relevant information is 0.01% of total population so can be ignored. These rows will be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.5</td>\n",
       "      <td>55.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.95</td>\n",
       "      <td>3.98</td>\n",
       "      <td>2.43</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.21</td>\n",
       "      <td>Premium</td>\n",
       "      <td>E</td>\n",
       "      <td>SI1</td>\n",
       "      <td>59.8</td>\n",
       "      <td>61.0</td>\n",
       "      <td>326</td>\n",
       "      <td>3.89</td>\n",
       "      <td>3.84</td>\n",
       "      <td>2.31</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.23</td>\n",
       "      <td>Good</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>56.9</td>\n",
       "      <td>65.0</td>\n",
       "      <td>327</td>\n",
       "      <td>4.05</td>\n",
       "      <td>4.07</td>\n",
       "      <td>2.31</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.29</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.4</td>\n",
       "      <td>58.0</td>\n",
       "      <td>334</td>\n",
       "      <td>4.20</td>\n",
       "      <td>4.23</td>\n",
       "      <td>2.63</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.31</td>\n",
       "      <td>Good</td>\n",
       "      <td>J</td>\n",
       "      <td>SI2</td>\n",
       "      <td>63.3</td>\n",
       "      <td>58.0</td>\n",
       "      <td>335</td>\n",
       "      <td>4.34</td>\n",
       "      <td>4.35</td>\n",
       "      <td>2.75</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53935</th>\n",
       "      <td>0.72</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>D</td>\n",
       "      <td>SI1</td>\n",
       "      <td>60.8</td>\n",
       "      <td>57.0</td>\n",
       "      <td>2757</td>\n",
       "      <td>5.75</td>\n",
       "      <td>5.76</td>\n",
       "      <td>3.50</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53936</th>\n",
       "      <td>0.72</td>\n",
       "      <td>Good</td>\n",
       "      <td>D</td>\n",
       "      <td>SI1</td>\n",
       "      <td>63.1</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2757</td>\n",
       "      <td>5.69</td>\n",
       "      <td>5.75</td>\n",
       "      <td>3.61</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53937</th>\n",
       "      <td>0.70</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>D</td>\n",
       "      <td>SI1</td>\n",
       "      <td>62.8</td>\n",
       "      <td>60.0</td>\n",
       "      <td>2757</td>\n",
       "      <td>5.66</td>\n",
       "      <td>5.68</td>\n",
       "      <td>3.56</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53938</th>\n",
       "      <td>0.86</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI2</td>\n",
       "      <td>61.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>2757</td>\n",
       "      <td>6.15</td>\n",
       "      <td>6.12</td>\n",
       "      <td>3.74</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53939</th>\n",
       "      <td>0.75</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>D</td>\n",
       "      <td>SI2</td>\n",
       "      <td>62.2</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2757</td>\n",
       "      <td>5.83</td>\n",
       "      <td>5.87</td>\n",
       "      <td>3.64</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>53932 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat        cut color clarity  depth  table  price     x     y     z  \\\n",
       "0       0.23      Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43   \n",
       "1       0.21    Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31   \n",
       "2       0.23       Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31   \n",
       "3       0.29    Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63   \n",
       "4       0.31       Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75   \n",
       "...      ...        ...   ...     ...    ...    ...    ...   ...   ...   ...   \n",
       "53935   0.72      Ideal     D     SI1   60.8   57.0   2757  5.75  5.76  3.50   \n",
       "53936   0.72       Good     D     SI1   63.1   55.0   2757  5.69  5.75  3.61   \n",
       "53937   0.70  Very Good     D     SI1   62.8   60.0   2757  5.66  5.68  3.56   \n",
       "53938   0.86    Premium     H     SI2   61.0   58.0   2757  6.15  6.12  3.74   \n",
       "53939   0.75      Ideal     D     SI2   62.2   55.0   2757  5.83  5.87  3.64   \n",
       "\n",
       "       clarity_rating  color_rating  \n",
       "0                   1             1  \n",
       "1                   2             1  \n",
       "2                   4             1  \n",
       "3                   3             5  \n",
       "4                   1             6  \n",
       "...               ...           ...  \n",
       "53935               2             0  \n",
       "53936               2             0  \n",
       "53937               2             0  \n",
       "53938               1             4  \n",
       "53939               1             0  \n",
       "\n",
       "[53932 rows x 12 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.dropna(inplace=True)\n",
    "diamonds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Checking for outliers\n",
    "Now we are going to revisit the summary table to check for outliers.\n",
    "\n",
    "**Use the `describe` method again and comment on what you see. After that, check if you have any outliers** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.797879</td>\n",
       "      <td>61.749336</td>\n",
       "      <td>57.457029</td>\n",
       "      <td>3932.136079</td>\n",
       "      <td>5.732007</td>\n",
       "      <td>5.735254</td>\n",
       "      <td>3.540266</td>\n",
       "      <td>3.051101</td>\n",
       "      <td>2.594211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.473986</td>\n",
       "      <td>1.432514</td>\n",
       "      <td>2.234064</td>\n",
       "      <td>3988.734835</td>\n",
       "      <td>1.119670</td>\n",
       "      <td>1.140343</td>\n",
       "      <td>0.702667</td>\n",
       "      <td>1.647109</td>\n",
       "      <td>1.701165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.200000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>326.000000</td>\n",
       "      <td>3.730000</td>\n",
       "      <td>3.680000</td>\n",
       "      <td>1.070000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.400000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>56.000000</td>\n",
       "      <td>949.750000</td>\n",
       "      <td>4.710000</td>\n",
       "      <td>4.720000</td>\n",
       "      <td>2.910000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.700000</td>\n",
       "      <td>61.800000</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>2401.000000</td>\n",
       "      <td>5.700000</td>\n",
       "      <td>5.710000</td>\n",
       "      <td>3.530000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.040000</td>\n",
       "      <td>62.500000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>5324.000000</td>\n",
       "      <td>6.540000</td>\n",
       "      <td>6.540000</td>\n",
       "      <td>4.040000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>5.010000</td>\n",
       "      <td>79.000000</td>\n",
       "      <td>95.000000</td>\n",
       "      <td>18823.000000</td>\n",
       "      <td>10.740000</td>\n",
       "      <td>58.900000</td>\n",
       "      <td>31.800000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              carat         depth         table         price             x  \\\n",
       "count  53932.000000  53932.000000  53932.000000  53932.000000  53932.000000   \n",
       "mean       0.797879     61.749336     57.457029   3932.136079      5.732007   \n",
       "std        0.473986      1.432514      2.234064   3988.734835      1.119670   \n",
       "min        0.200000     43.000000     43.000000    326.000000      3.730000   \n",
       "25%        0.400000     61.000000     56.000000    949.750000      4.710000   \n",
       "50%        0.700000     61.800000     57.000000   2401.000000      5.700000   \n",
       "75%        1.040000     62.500000     59.000000   5324.000000      6.540000   \n",
       "max        5.010000     79.000000     95.000000  18823.000000     10.740000   \n",
       "\n",
       "                  y             z  clarity_rating  color_rating  \n",
       "count  53932.000000  53932.000000    53932.000000  53932.000000  \n",
       "mean       5.735254      3.540266        3.051101      2.594211  \n",
       "std        1.140343      0.702667        1.647109      1.701165  \n",
       "min        3.680000      1.070000        0.000000      0.000000  \n",
       "25%        4.720000      2.910000        2.000000      1.000000  \n",
       "50%        5.710000      3.530000        3.000000      3.000000  \n",
       "75%        6.540000      4.040000        4.000000      4.000000  \n",
       "max       58.900000     31.800000        7.000000      6.000000  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "diamonds.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your comments here\n",
    "'''\n",
    "Maximum values of y & z look incorrect taking into account average values for these measurements and maximum value of table,\n",
    "which is calculated from x, y & z values.\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To manage these outliers, we are going to filter our DataFrame, we're going to take all the values that have a price higher than the 75th percentile.\n",
    "\n",
    "**Look for that quantile and filter the dataframe to clearly see the outliers. What do you think?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n",
    "# Loop below prints analysis vs quantile for each column in one go. But this information is excecive and hard to review, also\n",
    "# based on follow up questions it's been requested to analyse only 'price' column.\n",
    "\n",
    "# print(diamonds.quantile(0.75))\n",
    "# for col in diamonds.columns:\n",
    "#     if diamonds[col].dtype != 'object':\n",
    "#         print(col)\n",
    "#         print(diamonds.loc[diamonds[col] > diamonds[col].quantile(0.75)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12765</th>\n",
       "      <td>1.07</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>SI2</td>\n",
       "      <td>62.2</td>\n",
       "      <td>55.0</td>\n",
       "      <td>5325</td>\n",
       "      <td>6.54</td>\n",
       "      <td>6.58</td>\n",
       "      <td>4.08</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12766</th>\n",
       "      <td>1.11</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>H</td>\n",
       "      <td>SI1</td>\n",
       "      <td>60.6</td>\n",
       "      <td>59.0</td>\n",
       "      <td>5325</td>\n",
       "      <td>6.75</td>\n",
       "      <td>6.71</td>\n",
       "      <td>4.08</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12767</th>\n",
       "      <td>1.14</td>\n",
       "      <td>Premium</td>\n",
       "      <td>G</td>\n",
       "      <td>SI1</td>\n",
       "      <td>62.2</td>\n",
       "      <td>58.0</td>\n",
       "      <td>5326</td>\n",
       "      <td>6.64</td>\n",
       "      <td>6.70</td>\n",
       "      <td>4.15</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12768</th>\n",
       "      <td>1.07</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.1</td>\n",
       "      <td>59.0</td>\n",
       "      <td>5327</td>\n",
       "      <td>6.52</td>\n",
       "      <td>6.56</td>\n",
       "      <td>4.06</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12769</th>\n",
       "      <td>1.33</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>J</td>\n",
       "      <td>SI1</td>\n",
       "      <td>64.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>5327</td>\n",
       "      <td>6.93</td>\n",
       "      <td>7.01</td>\n",
       "      <td>4.46</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27745</th>\n",
       "      <td>2.00</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>H</td>\n",
       "      <td>SI1</td>\n",
       "      <td>62.8</td>\n",
       "      <td>57.0</td>\n",
       "      <td>18803</td>\n",
       "      <td>7.95</td>\n",
       "      <td>8.00</td>\n",
       "      <td>5.01</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27746</th>\n",
       "      <td>2.07</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>G</td>\n",
       "      <td>SI2</td>\n",
       "      <td>62.5</td>\n",
       "      <td>55.0</td>\n",
       "      <td>18804</td>\n",
       "      <td>8.20</td>\n",
       "      <td>8.13</td>\n",
       "      <td>5.11</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27747</th>\n",
       "      <td>1.51</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>G</td>\n",
       "      <td>IF</td>\n",
       "      <td>61.7</td>\n",
       "      <td>55.0</td>\n",
       "      <td>18806</td>\n",
       "      <td>7.37</td>\n",
       "      <td>7.41</td>\n",
       "      <td>4.56</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27748</th>\n",
       "      <td>2.00</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>G</td>\n",
       "      <td>SI1</td>\n",
       "      <td>63.5</td>\n",
       "      <td>56.0</td>\n",
       "      <td>18818</td>\n",
       "      <td>7.90</td>\n",
       "      <td>7.97</td>\n",
       "      <td>5.04</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27749</th>\n",
       "      <td>2.29</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>VS2</td>\n",
       "      <td>60.8</td>\n",
       "      <td>60.0</td>\n",
       "      <td>18823</td>\n",
       "      <td>8.50</td>\n",
       "      <td>8.47</td>\n",
       "      <td>5.16</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13481 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat        cut color clarity  depth  table  price     x     y     z  \\\n",
       "12765   1.07      Ideal     E     SI2   62.2   55.0   5325  6.54  6.58  4.08   \n",
       "12766   1.11      Ideal     H     SI1   60.6   59.0   5325  6.75  6.71  4.08   \n",
       "12767   1.14    Premium     G     SI1   62.2   58.0   5326  6.64  6.70  4.15   \n",
       "12768   1.07    Premium     H     VS2   62.1   59.0   5327  6.52  6.56  4.06   \n",
       "12769   1.33  Very Good     J     SI1   64.0   53.0   5327  6.93  7.01  4.46   \n",
       "...      ...        ...   ...     ...    ...    ...    ...   ...   ...   ...   \n",
       "27745   2.00  Very Good     H     SI1   62.8   57.0  18803  7.95  8.00  5.01   \n",
       "27746   2.07      Ideal     G     SI2   62.5   55.0  18804  8.20  8.13  5.11   \n",
       "27747   1.51      Ideal     G      IF   61.7   55.0  18806  7.37  7.41  4.56   \n",
       "27748   2.00  Very Good     G     SI1   63.5   56.0  18818  7.90  7.97  5.04   \n",
       "27749   2.29    Premium     I     VS2   60.8   60.0  18823  8.50  8.47  5.16   \n",
       "\n",
       "       clarity_rating  color_rating  \n",
       "12765               1             1  \n",
       "12766               2             4  \n",
       "12767               2             3  \n",
       "12768               3             4  \n",
       "12769               2             6  \n",
       "...               ...           ...  \n",
       "27745               2             4  \n",
       "27746               1             3  \n",
       "27747               7             3  \n",
       "27748               2             3  \n",
       "27749               3             5  \n",
       "\n",
       "[13481 rows x 12 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.loc[diamonds['price'] > diamonds['price'].quantile(0.75)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset is really big and the outliers are really far apart from the rest of the values. To see this more clearly we will use a boxplot, which plots the median, 25th and 75th quartile, the maximum and minimum, as well as any outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAEhCAYAAACuva4PAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAXBUlEQVR4nO3dfYxdd53f8ffXM2lxYCnxeOJaTlLvEqu7aGmc1SiiCqzy4NnOpgvJVspqI4pnRYRX4sEBoW7TVUm2SF2hapddTCtaL0m51rLsujwoBpkhY5cHIa1SxqwbhzrFFjJ5wNjDBUrSPJA78+0fc0Id43Fm7Dm/c+6d90sanXvOvcf3k8g6/pzfOXN+kZlIkiSpfmuaDiBJkrRaWLwkSZIKsXhJkiQVYvGSJEkqxOIlSZJUiMVLkiSpkOGmAyzF+vXrc/PmzU3HkFadQ4cO/SAzR5vOoQvjsVNqxvmOnX1RvDZv3szMzEzTMaRVJyK+23QGXTiPnVIzznfs9FKjJElSIRYvSZKkQmotXhHxmoj4dEQ8GhFHI+KfRsS6iJiOiGPV8rI6M0iSJLVF3SNeHwGmMvOXgWuAo8DdwMHM3AIcrNYlSZIGXm3FKyJeDfw6cB9AZv40M38M3Ap0qo91gNvqyiBJktQmdY54/RIwC/zXiPi7iPh4RLwS2JCZJwGq5eXn2jkidkTETETMzM7O1hhTkiSpjDqL1zDwa8DHMvNa4P+yjMuKmbk7M8cyc2x01McI9atut8vOnTvpdrtNR5GkvuBxc7DVWbyeAJ7IzIeq9U+zUMRORcRGgGp5usYMalin0+HIkSPs2bOn6SiS1Bc8bg622opXZn4feDwi/nG16WbgfwH7gMlq2yTwQF0Z1Kxut8vU1BSZydTUlGdvkvQyPG4Ovrp/q/E9wCcj4mFgK/DHwIeA8Yg4BoxX6xpAnU6H+fl5AObm5jx7k6SX4XFz8NVavDLzcHWf1j/JzNsy80eZ2c3MmzNzS7X8YZ0Z1JwDBw7Q6/UA6PV6TE9PN5xIktrN4+bg88n1qs22bdsYHl6YDnR4eJjx8fGGE0lSu3ncHHwWL9VmcnKSNWsW/ooNDQ2xffv2hhNJUrt53Bx8Fi/VZmRkhImJCSKCiYkJRkZGmo4kSa3mcXPwDTcdQINtcnKSEydOeNYmSUvkcXOwWbxUq5GREXbt2tV0DEnqGx43B5uXGiVJkgqxeEmSJBVi8ZIkSSrE4iVJklSIxUuSJKkQi5cktUhEvCIi/kdE/M+I+FZE/Ltq+7qImI6IY9XysqazSlo+i5cktcvzwE2ZeQ2wFZiIiDcAdwMHM3MLcLBal9RnLF6S1CK54Olq9ZLqJ4FbgU61vQPc1kA8SRfJ4iVJLRMRQxFxGDgNTGfmQ8CGzDwJUC0vX2TfHRExExEzs7Oz5UJLWhKLlyS1TGbOZeZW4Arguoj41WXsuzszxzJzbHR0tL6Qki6IxUuSWiozfwx8BZgATkXERoBqebrBaJIukMVLklokIkYj4jXV67XANuBRYB8wWX1sEnigmYSSLoaTZEtSu2wEOhExxMLJ8d7M/EJE/C2wNyLuBB4Dbm8ypKQLY/GSpBbJzIeBa8+xvQvcXD6RpJXkpUZJkqRCLF6SJEmFWLwkSZIKsXhJkiQVYvGSJEkqxOIlSZJUiMVLkiSpEIuXJElSIRYvSZKkQixekiRJhVi8JEmSCrF4SZIkFWLxkiRJKsTiJUmSVIjFS5IkqRCLlyRJUiHDdf7hEXECeAqYA3qZORYR64C/ATYDJ4Dfycwf1ZlDkiSpDUqMeN2YmVszc6xavxs4mJlbgIPVuiRJ0sBr4lLjrUCnet0BbmsggyRJUnF1F68EHoyIQxGxo9q2ITNPAlTLy2vOIEmS1Aq13uMFXJ+Z34uIy4HpiHh0qTtWRW0HwFVXXVVXPkmSpGJqHfHKzO9Vy9PA54DrgFMRsRGgWp5eZN/dmTmWmWOjo6N1xpQkSSqituIVEa+MiF948TXwG8AjwD5gsvrYJPBAXRkkSZLapM5LjRuAz0XEi9/zV5k5FRHfAPZGxJ3AY8DtNWaQJElqjdqKV2Z+B7jmHNu7wM11fa8kSVJb+eR6SZKkQixekiRJhVi8JEmSCrF4SZIkFWLxkiRJKsTiJUmSVIjFS5JaJCKujIgvR8TRiPhWRNxVbf+jiHgyIg5XP7c0nVXS8tU9V6MkaXl6wPsz85vV7B+HImK6eu/PMvNPGswm6SJZvCSpRTLzJHCyev1URBwFNjWbStJK8VKjJLVURGwGrgUeqja9OyIejoj7I+KyxoJJumAWL0lqoYh4FfAZ4L2Z+RPgY8Brga0sjIj96SL77YiImYiYmZ2dLZZX0tJYvCSpZSLiEhZK1ycz87MAmXkqM+cycx74C+C6c+2bmbszcywzx0ZHR8uFlrQkFi9JapGICOA+4GhmfviM7RvP+NhvA4+Uzibp4nlzvSS1y/XA24AjEXG42vaHwB0RsRVI4ATw+83Ek3QxLF6S1CKZ+XUgzvHW/tJZJK08LzVKkiQVYvGSJEkqxOIlSZJUiMVLkiSpEIuXJElSIRYvSZKkQixekiRJhVi8JEmSCrF4SZIkFWLxkiRJKsTiJUmSVIjFS5IkqRCLlyRJUiEWL0mSpEIsXpIkSYVYvCRJkgqxeEmSJBVi8ZIkSSrE4iVJklSIxUuSJKkQi5ckSVIhtReviBiKiL+LiC9U6+siYjoijlXLy+rOIEmS1AYlRrzuAo6esX43cDAztwAHq3VJkqSBV2vxiogrgH8OfPyMzbcCnep1B7itzgySJEltUfeI158DfwDMn7FtQ2aeBKiWl59rx4jYEREzETEzOztbc0xJkqT61Va8IuK3gNOZeehC9s/M3Zk5lpljo6OjK5xOkiSpvOEa/+zrgbdExC3AK4BXR8RfAqciYmNmnoyIjcDpGjNIkiS1Rm0jXpn5bzLziszcDPwu8N8z818C+4DJ6mOTwAN1ZZAkSWqTJp7j9SFgPCKOAePVuiRJ0sCr81Ljz2TmV4CvVK+7wM0lvleSJKlNfHK9JElSIRYvSWqRiLgyIr4cEUcj4lsRcVe13Vk/pAFg8ZKkdukB78/MXwHeALwrIl6Hs35IA8HiJUktkpknM/Ob1eunWJhybRPO+iENBIuXJLVURGwGrgUeYomzfkhqN4uXJLVQRLwK+Azw3sz8yTL2c7o1qcUsXpLUMhFxCQul65OZ+dlq86lqtg/ON+uH061J7WbxkqQWiYgA7gOOZuaHz3jLWT+kAVDkAaqSpCW7HngbcCQiDlfb/pCFWT72RsSdwGPA7Q3lk3QRLF6S1CKZ+XUgFnnbWT+kPuelRkmSpEIsXpIkSYVYvCRJkgqxeEmSJBVi8ZIkSSrE4iVJklSIxUuSJKkQi5ckSVIhFi9JkqRCLF6SJEmFWLwkSZIKsXhJkiQVYvGSJEkqxOIlSZJUiMVLkiSpEIuXJElSIRYvSZKkQixeqlW322Xnzp10u92mo0iS1DiLl2rV6XQ4cuQIe/bsaTqKJEmNs3ipNt1ul6mpKTKTqakpR70kSauexUu16XQ6zM/PAzA3N+eolyRp1bN4qTYHDhyg1+sB0Ov1mJ6ebjiRJEnNetniFRHvjojLSoTRYNm2bRvDw8MADA8PMz4+3nAiSZKatZQRr38IfCMi9kbERERE3aE0GCYnJ1mzZuGv2NDQENu3b284kVRORByMiFvO2ra7qTyS2uFli1dm/ltgC3Af8HvAsYj444h4bc3Z1OdGRkaYmJggIpiYmGBkZKTpSFJJvwj864i494xtY02FkdQOS7rHKzMT+H710wMuAz4dEf+hxmwaAJOTk7z+9a93tEur0Y+Bm4ENEfH5iPgHTQeS1Lzhl/tAROwEJoEfAB8H/lVmvhARa4BjwB8sst8rgK8Bf7/6nk9n5r0RsQ74G2AzcAL4ncz80cX/p6iNRkZG2LVrV9MxpCZEZvaAd0bE7wFfZ+GkVdIqtpQRr/XAv8jMf5aZ/y0zXwDIzHngt86z3/PATZl5DbAVmIiINwB3AwczcwtwsFqXpEHzn198kZmfYOFWjQebCiOpHZZyj9c9mfndRd47ep79MjOfrlYvqX4SuBXoVNs7wG3LSixJfSAz/8tZ64cy8+1N5ZHUDrU+xysihiLiMHAamM7Mh4ANmXkSoFpevsi+OyJiJiJmZmdn64wpSZJURK3FKzPnMnMrcAVwXUT86jL23Z2ZY5k5Njo6Wl9ISZKkQoo8uT4zfwx8BZgATkXERoBqebpEBkmSpKbVVrwiYjQiXlO9XgtsAx4F9rHwW5JUywfqyiBJ/SYi7o+I0xHxyBnb/iginoyIw9XPLef7MyS118s+TuIibAQ6ETHEQsHbm5lfiIi/BfZGxJ3AY8DtNWaQpH7zCeA/AmfPKv9nmfkn5eNIWkm1Fa/MfBi49hzbuyw8VFCSdJbM/FpEbG46h6R6FLnHS5J00d4dEQ9XlyIXfRCrvxEutZvFS5La72PAa1l4GPVJ4E8X+6C/ES61m8VLklouM09Vj+eZB/4CuK7pTJIujMVLklruxUfwVH4beGSxz0pqtzp/q1GStEwR8SngBmB9RDwB3AvcEBFbWZh27QTw+40FlHRRLF6S1CKZecc5Nt9XPIikWnipUZIkqRCLlyRJUiEWL0mSpEIsXpIkSYVYvCRJkgqxeEmSJBVi8ZIkSSrE4iVJklSIxUuSJKkQi5ckSVIhFi9JkqRCLF6SJEmFWLwkSZIKsXhJkiQVYvGSJEkqxOIlSZJUiMVLkiSpEIuXJElSIRYvSZKkQixekiRJhVi8JEmSCrF4SZIkFWLxkiRJKsTiJUmSVIjFS5IkqRCLlyRJUiEWL0mSpEIsXpIkSYVYvCRJkgqxeEmSJBVSW/GKiCsj4ssRcTQivhURd1Xb10XEdEQcq5aX1ZVBkiSpTeoc8eoB78/MXwHeALwrIl4H3A0czMwtwMFqXZIERMT9EXE6Ih45Y5snrNKAqK14ZebJzPxm9fop4CiwCbgV6FQf6wC31ZVBkvrQJ4CJs7Z5wioNiCL3eEXEZuBa4CFgQ2aehIVyBly+yD47ImImImZmZ2dLxJSkxmXm14AfnrXZE1ZpQNRevCLiVcBngPdm5k+Wul9m7s7MscwcGx0drS+gJLXfkk5YwZNWqe1qLV4RcQkLpeuTmfnZavOpiNhYvb8ROF1nBklaTTxpldqtzt9qDOA+4GhmfviMt/YBk9XrSeCBujJI0oDwhFUaEHWOeF0PvA24KSIOVz+3AB8CxiPiGDBerUuSFucJqzQghuv6gzPz60As8vbNdX2vJPWziPgUcAOwPiKeAO5l4QR1b0TcCTwG3N5cQkkXo7biJUlavsy8Y5G3PGGVBoBTBkmSJBVi8ZIkSSrE4iVJklSIxUuSJKkQi5ckSVIhFi9JkqRCLF6SJLVIt9tl586ddLvdpqOoBhYvSZJapNPpcOTIEfbs2dN0FNXA4iVJUkt0u12mpqbITKamphz1GkAWL0mSWqLT6TA/Pw/A3Nyco14DyOIlSVJLHDhwgF6vB0Cv12N6errhRFppFi9Jklpi27ZtDA8vTKM8PDzM+Ph4w4m00ixekiS1xOTkJGvWLPzTPDQ0xPbt2xtOpJVm8ZIkqSVGRkaYmJggIpiYmGBkZKTpSFphFi9JklrkLW95C5deeilvfvObm46iGli8JElqkX379vHMM8/w+c9/vukoqoHFS5KklvA5XoPP4iVJUkt0Oh3m5uaAhcdJ+ByvwWPxkiSpJQ4cOPCz4jU3N+dzvAaQxUuSpJZ44xvf+JL1N73pTQ0lUV0sXpIktURENB1BNbN4SZLUEl/96lfPu67+Z/GSJKmlHAEbPBYvSZJa4rnnnnvJ+rPPPttQEtXF4iVJklSIxUuSJKkQi5ckSVIhw00HkCSpTh/96Ec5fvx40zEu2F133dV0hCW5+uqrec973tN0jNZzxEuSJKkQR7wkSQOtn0Zhbrjhhp/b9pGPfKR8ENXGES9JklrirW9960vWt2/f3lAS1cXipVp1u1127txJt9ttOorU9yLiREQciYjDETHTdB6tvHe84x0vWX/729/eUBLVxeKlWnU6HY4cOcKePXuajiINihszc2tmjjUdRPVYv3494GjXoLJ4qTbdbpcvfvGLZCb79+931EuSlmDTpk1cc801jnYNqNqKV0TcHxGnI+KRM7ati4jpiDhWLS+r6/vVvE6nwwsvvADACy+84KiXdPESeDAiDkXEjnN9ICJ2RMRMRMzMzs4Wjifp5dQ54vUJYOKsbXcDBzNzC3CwWteAevDBB1+y/qUvfamhJNLAuD4zfw34TeBdEfHrZ38gM3dn5lhmjo2OjpZPKOm8aitemfk14Idnbb4V6FSvO8BtdX2/2icimo4g9bXM/F61PA18Driu2USSlqv0PV4bMvMkQLW8vPD3q6DnnnvuJevPPvtsQ0mk/hcRr4yIX3jxNfAbwCPn30tS27T2AarV/Qs7AK666qqG00hS4zYAn6tGjoeBv8rMqWYjSVqu0sXrVERszMyTEbEROL3YBzNzN7AbYGxsLEsFlKQ2yszvANc0nUPSxSldvPYBk8CHquUDhb+/r/X7RK/QH5O9OtGrJKkutRWviPgUcAOwPiKeAO5loXDtjYg7gceA2+v6fknSyhuEE8C2e/H/bz+cqPa7Jk60aytemXnHIm/dXNd3Drp+G4W58cYbyfz/V4kjwslepT53/PhxDj9ylLlL1zUdZWCt+enCcfPQd041nGSwDT1z9oMXymjtzfXqfx/4wAf44Ac/+LP1e+65p8E0klbK3KXrePaXb2k6hnRR1j66v5Hvdcog1eamm2762euI4MYbb2wwjSRJzXPES7W68sorefzxxx3tkgbEk08+ydAz/6ex0QJppQw90+XJJ3vFv9fipVqtW7eOdevWOdolSRIWL0nSMmzatInvPz/sPV7qe2sf3c+mTRuKf6/3eEmSJBVi8ZIkSSrES42SpGUZeuaH3lxfozXP/QSA+Ve8uuEkg23hOV7lLzVavCRJS3b11Vc3HWHgHT/+FABX/1L5UrC6bGjk7/OqLV5Oe1GGU1+U4fySKsW/Z/V78XjpTB+DadUWL6e9KMOpL+rX1LQXkqTlW7XFC5z2QoPBe20kqX/4W42SJEmFrNoRL6e90KBoatoLSdLyOeIlSZJUyKod8XLaCw2Kpqa9kCQtnyNekiRJhazaES/w6csl+ATm+jX19GVJ0vKt2uLl05fL8AnMJTTz9GVJ0vKt2uLl05fL8AnMkprWbzOVfPvb3+b555/nne98J5dccknTcZbMGTSWxnu8JElqkfn5eebn5zl1yhk/BtGqHfGSJK0O/TQK0+12ueOOOwB4+umnueeeexgZGWk4lVaSxauP9NtwOfTnJNkOl0tqSqfTYX5+HoC5uTn27NnD+973voZTaSV5qVG1Wrt2LWvXrm06hiT1hQMHDtDrLcxE0ev1mJ6ebjiRVpojXn3EURhJGmzbtm1j//799Ho9hoeHGR8fbzqSVpgjXpIktcTk5CRr1iz80zw0NMT27dsbTqSVZvGSJKklRkZGmJiYICKYmJjwxvoB5KVGSZJaZHJykhMnTjjaNaAc8ZKkPhERExHxvyPieETc3XQe1WNkZIRdu3Y52jWgLF6S1AciYgj4T8BvAq8D7oiI1zWbStJyWbwkqT9cBxzPzO9k5k+BvwZubTiTpGWyeElSf9gEPH7G+hPVNkl9xOIlSf0hzrEtf+5DETsiYiYiZmZnZwvEkrQcFi9J6g9PAFeesX4F8L2zP5SZuzNzLDPHRkdHi4WTtDQWL0nqD98AtkTEL0bE3wN+F9jXcCZJyxSZPzdS3ToRMQt8t+kcumDrgR80HUIX5B9lpsMmLRERtwB/DgwB92fmv3+Zz3vs7F8eN/vbosfOvihe6m8RMZOZY03nkKR+4XFzcHmpUZIkqRCLlyRJUiEWL5Wwu+kAktRnPG4OKO/xkiRJKsQRL0mSpEIsXpIkSYVYvCRJkgqxeEmSJBVi8ZIkSSrk/wFDVZRJhL8pUQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Run this code\n",
    "fig, ax = plt.subplots(1,2, figsize=(10, 5))\n",
    "sns.boxplot(y=diamonds.y, ax=ax[0])\n",
    "sns.boxplot(y=diamonds.z, ax=ax[1])\n",
    "plt.subplots_adjust(wspace=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can see that all the values are within an acceptable range, but we have 2 big outliers in y and 1 in z. Now we know that our max values for y should be around 10 and the values for z should be around 6, so let's filter our dataset to find values higher than 10 in it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24067</th>\n",
       "      <td>2.00</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI2</td>\n",
       "      <td>58.9</td>\n",
       "      <td>57.0</td>\n",
       "      <td>12210</td>\n",
       "      <td>8.09</td>\n",
       "      <td>58.90</td>\n",
       "      <td>8.06</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25998</th>\n",
       "      <td>4.01</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>I1</td>\n",
       "      <td>61.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>15223</td>\n",
       "      <td>10.14</td>\n",
       "      <td>10.10</td>\n",
       "      <td>6.17</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27415</th>\n",
       "      <td>5.01</td>\n",
       "      <td>Fair</td>\n",
       "      <td>J</td>\n",
       "      <td>I1</td>\n",
       "      <td>65.5</td>\n",
       "      <td>59.0</td>\n",
       "      <td>18018</td>\n",
       "      <td>10.74</td>\n",
       "      <td>10.54</td>\n",
       "      <td>6.98</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27630</th>\n",
       "      <td>4.50</td>\n",
       "      <td>Fair</td>\n",
       "      <td>J</td>\n",
       "      <td>I1</td>\n",
       "      <td>65.8</td>\n",
       "      <td>58.0</td>\n",
       "      <td>18531</td>\n",
       "      <td>10.23</td>\n",
       "      <td>10.16</td>\n",
       "      <td>6.72</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49189</th>\n",
       "      <td>0.51</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>61.8</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2075</td>\n",
       "      <td>5.15</td>\n",
       "      <td>31.80</td>\n",
       "      <td>5.12</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat      cut color clarity  depth  table  price      x      y     z  \\\n",
       "24067   2.00  Premium     H     SI2   58.9   57.0  12210   8.09  58.90  8.06   \n",
       "25998   4.01  Premium     I      I1   61.0   61.0  15223  10.14  10.10  6.17   \n",
       "27415   5.01     Fair     J      I1   65.5   59.0  18018  10.74  10.54  6.98   \n",
       "27630   4.50     Fair     J      I1   65.8   58.0  18531  10.23  10.16  6.72   \n",
       "49189   0.51    Ideal     E     VS1   61.8   55.0   2075   5.15  31.80  5.12   \n",
       "\n",
       "       clarity_rating  color_rating  \n",
       "24067               1             4  \n",
       "25998               0             5  \n",
       "27415               0             6  \n",
       "27630               0             6  \n",
       "49189               4             1  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "diamonds.loc[diamonds['y'] > 10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>23644</th>\n",
       "      <td>3.65</td>\n",
       "      <td>Fair</td>\n",
       "      <td>H</td>\n",
       "      <td>I1</td>\n",
       "      <td>67.1</td>\n",
       "      <td>53.0</td>\n",
       "      <td>11668</td>\n",
       "      <td>9.53</td>\n",
       "      <td>9.48</td>\n",
       "      <td>6.38</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24067</th>\n",
       "      <td>2.00</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI2</td>\n",
       "      <td>58.9</td>\n",
       "      <td>57.0</td>\n",
       "      <td>12210</td>\n",
       "      <td>8.09</td>\n",
       "      <td>58.90</td>\n",
       "      <td>8.06</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24328</th>\n",
       "      <td>3.50</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>H</td>\n",
       "      <td>I1</td>\n",
       "      <td>62.8</td>\n",
       "      <td>57.0</td>\n",
       "      <td>12587</td>\n",
       "      <td>9.65</td>\n",
       "      <td>9.59</td>\n",
       "      <td>6.03</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25998</th>\n",
       "      <td>4.01</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>I1</td>\n",
       "      <td>61.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>15223</td>\n",
       "      <td>10.14</td>\n",
       "      <td>10.10</td>\n",
       "      <td>6.17</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25999</th>\n",
       "      <td>4.01</td>\n",
       "      <td>Premium</td>\n",
       "      <td>J</td>\n",
       "      <td>I1</td>\n",
       "      <td>62.5</td>\n",
       "      <td>62.0</td>\n",
       "      <td>15223</td>\n",
       "      <td>10.02</td>\n",
       "      <td>9.94</td>\n",
       "      <td>6.24</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26194</th>\n",
       "      <td>2.01</td>\n",
       "      <td>Fair</td>\n",
       "      <td>G</td>\n",
       "      <td>SI2</td>\n",
       "      <td>65.6</td>\n",
       "      <td>56.0</td>\n",
       "      <td>15562</td>\n",
       "      <td>7.89</td>\n",
       "      <td>7.84</td>\n",
       "      <td>6.16</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26431</th>\n",
       "      <td>3.40</td>\n",
       "      <td>Fair</td>\n",
       "      <td>D</td>\n",
       "      <td>I1</td>\n",
       "      <td>66.8</td>\n",
       "      <td>52.0</td>\n",
       "      <td>15964</td>\n",
       "      <td>9.42</td>\n",
       "      <td>9.34</td>\n",
       "      <td>6.27</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26444</th>\n",
       "      <td>4.00</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>I</td>\n",
       "      <td>I1</td>\n",
       "      <td>63.3</td>\n",
       "      <td>58.0</td>\n",
       "      <td>15984</td>\n",
       "      <td>10.01</td>\n",
       "      <td>9.94</td>\n",
       "      <td>6.31</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26534</th>\n",
       "      <td>3.67</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>I1</td>\n",
       "      <td>62.4</td>\n",
       "      <td>56.0</td>\n",
       "      <td>16193</td>\n",
       "      <td>9.86</td>\n",
       "      <td>9.81</td>\n",
       "      <td>6.13</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27130</th>\n",
       "      <td>4.13</td>\n",
       "      <td>Fair</td>\n",
       "      <td>H</td>\n",
       "      <td>I1</td>\n",
       "      <td>64.8</td>\n",
       "      <td>61.0</td>\n",
       "      <td>17329</td>\n",
       "      <td>10.00</td>\n",
       "      <td>9.85</td>\n",
       "      <td>6.43</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27415</th>\n",
       "      <td>5.01</td>\n",
       "      <td>Fair</td>\n",
       "      <td>J</td>\n",
       "      <td>I1</td>\n",
       "      <td>65.5</td>\n",
       "      <td>59.0</td>\n",
       "      <td>18018</td>\n",
       "      <td>10.74</td>\n",
       "      <td>10.54</td>\n",
       "      <td>6.98</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27630</th>\n",
       "      <td>4.50</td>\n",
       "      <td>Fair</td>\n",
       "      <td>J</td>\n",
       "      <td>I1</td>\n",
       "      <td>65.8</td>\n",
       "      <td>58.0</td>\n",
       "      <td>18531</td>\n",
       "      <td>10.23</td>\n",
       "      <td>10.16</td>\n",
       "      <td>6.72</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27679</th>\n",
       "      <td>3.51</td>\n",
       "      <td>Premium</td>\n",
       "      <td>J</td>\n",
       "      <td>VS2</td>\n",
       "      <td>62.5</td>\n",
       "      <td>59.0</td>\n",
       "      <td>18701</td>\n",
       "      <td>9.66</td>\n",
       "      <td>9.63</td>\n",
       "      <td>6.03</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48410</th>\n",
       "      <td>0.51</td>\n",
       "      <td>Very Good</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>61.8</td>\n",
       "      <td>54.7</td>\n",
       "      <td>1970</td>\n",
       "      <td>5.12</td>\n",
       "      <td>5.15</td>\n",
       "      <td>31.80</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat        cut color clarity  depth  table  price      x      y  \\\n",
       "23644   3.65       Fair     H      I1   67.1   53.0  11668   9.53   9.48   \n",
       "24067   2.00    Premium     H     SI2   58.9   57.0  12210   8.09  58.90   \n",
       "24328   3.50      Ideal     H      I1   62.8   57.0  12587   9.65   9.59   \n",
       "25998   4.01    Premium     I      I1   61.0   61.0  15223  10.14  10.10   \n",
       "25999   4.01    Premium     J      I1   62.5   62.0  15223  10.02   9.94   \n",
       "26194   2.01       Fair     G     SI2   65.6   56.0  15562   7.89   7.84   \n",
       "26431   3.40       Fair     D      I1   66.8   52.0  15964   9.42   9.34   \n",
       "26444   4.00  Very Good     I      I1   63.3   58.0  15984  10.01   9.94   \n",
       "26534   3.67    Premium     I      I1   62.4   56.0  16193   9.86   9.81   \n",
       "27130   4.13       Fair     H      I1   64.8   61.0  17329  10.00   9.85   \n",
       "27415   5.01       Fair     J      I1   65.5   59.0  18018  10.74  10.54   \n",
       "27630   4.50       Fair     J      I1   65.8   58.0  18531  10.23  10.16   \n",
       "27679   3.51    Premium     J     VS2   62.5   59.0  18701   9.66   9.63   \n",
       "48410   0.51  Very Good     E     VS1   61.8   54.7   1970   5.12   5.15   \n",
       "\n",
       "           z  clarity_rating  color_rating  \n",
       "23644   6.38               0             4  \n",
       "24067   8.06               1             4  \n",
       "24328   6.03               0             4  \n",
       "25998   6.17               0             5  \n",
       "25999   6.24               0             6  \n",
       "26194   6.16               1             3  \n",
       "26431   6.27               0             0  \n",
       "26444   6.31               0             5  \n",
       "26534   6.13               0             5  \n",
       "27130   6.43               0             4  \n",
       "27415   6.98               0             6  \n",
       "27630   6.72               0             6  \n",
       "27679   6.03               3             6  \n",
       "48410  31.80               4             1  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.loc[diamonds['z'] > 6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have found the outlier, let's use the function we defined earlier to correct this value. First, we need to change the value to 0 (because that's how we defined the function before) and then we will apply it.\n",
    "\n",
    "**Apply `calculate_z` for the row with the outlier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n",
    "\n",
    "diamonds['z'] = np.where(diamonds['z'] > 6, diamonds.apply(lambda row: calculate_z(row), axis=1), diamonds['z'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check if we actually corrected the outlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "carat                  0.51\n",
       "cut               Very Good\n",
       "color                     E\n",
       "clarity                 VS1\n",
       "depth                  61.8\n",
       "table                  54.7\n",
       "price                  1970\n",
       "x                      5.12\n",
       "y                      5.15\n",
       "z                      3.17\n",
       "clarity_rating            4\n",
       "color_rating              1\n",
       "Name: 48410, dtype: object"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.loc[48410]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cool! Now let's validate our new `z`. We will check if we obtain the same value of depth using our validate function. If the formula applies, this means could approximate the real value of `z`.\n",
    "\n",
    "**Apply `validate_z` to the row used earlier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "      <th>depth_calc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "      <td>53932.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.797879</td>\n",
       "      <td>61.749336</td>\n",
       "      <td>57.457029</td>\n",
       "      <td>3932.136079</td>\n",
       "      <td>5.732007</td>\n",
       "      <td>5.735254</td>\n",
       "      <td>3.539933</td>\n",
       "      <td>3.051101</td>\n",
       "      <td>2.594211</td>\n",
       "      <td>61.748650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.473986</td>\n",
       "      <td>1.432514</td>\n",
       "      <td>2.234064</td>\n",
       "      <td>3988.734835</td>\n",
       "      <td>1.119670</td>\n",
       "      <td>1.140343</td>\n",
       "      <td>0.695224</td>\n",
       "      <td>1.647109</td>\n",
       "      <td>1.701165</td>\n",
       "      <td>1.494469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.200000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>326.000000</td>\n",
       "      <td>3.730000</td>\n",
       "      <td>3.680000</td>\n",
       "      <td>1.070000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>16.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.400000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>56.000000</td>\n",
       "      <td>949.750000</td>\n",
       "      <td>4.710000</td>\n",
       "      <td>4.720000</td>\n",
       "      <td>2.910000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>61.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.700000</td>\n",
       "      <td>61.800000</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>2401.000000</td>\n",
       "      <td>5.700000</td>\n",
       "      <td>5.710000</td>\n",
       "      <td>3.530000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>61.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.040000</td>\n",
       "      <td>62.500000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>5324.000000</td>\n",
       "      <td>6.540000</td>\n",
       "      <td>6.540000</td>\n",
       "      <td>4.040000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>62.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>5.010000</td>\n",
       "      <td>79.000000</td>\n",
       "      <td>95.000000</td>\n",
       "      <td>18823.000000</td>\n",
       "      <td>10.740000</td>\n",
       "      <td>58.900000</td>\n",
       "      <td>19.730000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>100.700000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              carat         depth         table         price             x  \\\n",
       "count  53932.000000  53932.000000  53932.000000  53932.000000  53932.000000   \n",
       "mean       0.797879     61.749336     57.457029   3932.136079      5.732007   \n",
       "std        0.473986      1.432514      2.234064   3988.734835      1.119670   \n",
       "min        0.200000     43.000000     43.000000    326.000000      3.730000   \n",
       "25%        0.400000     61.000000     56.000000    949.750000      4.710000   \n",
       "50%        0.700000     61.800000     57.000000   2401.000000      5.700000   \n",
       "75%        1.040000     62.500000     59.000000   5324.000000      6.540000   \n",
       "max        5.010000     79.000000     95.000000  18823.000000     10.740000   \n",
       "\n",
       "                  y             z  clarity_rating  color_rating    depth_calc  \n",
       "count  53932.000000  53932.000000    53932.000000  53932.000000  53932.000000  \n",
       "mean       5.735254      3.539933        3.051101      2.594211     61.748650  \n",
       "std        1.140343      0.695224        1.647109      1.701165      1.494469  \n",
       "min        3.680000      1.070000        0.000000      0.000000     16.100000  \n",
       "25%        4.720000      2.910000        2.000000      1.000000     61.000000  \n",
       "50%        5.710000      3.530000        3.000000      3.000000     61.800000  \n",
       "75%        6.540000      4.040000        4.000000      4.000000     62.500000  \n",
       "max       58.900000     19.730000        7.000000      6.000000    100.700000  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "diamonds['depth_calc'] = diamonds.apply(lambda row: calculate_depth(row), axis=1)\n",
    "diamonds.describe()\n",
    "# There are still some funny numbers in min and max but mean, quartiles and st.deviation are almost the same so let's leave\n",
    "# original depth unchanged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "diamonds.drop(['depth_calc'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's do the same for `y`. First, let's filter the DataFrame to find the outliers. We said that the maximum values should be around 10, so let's check what are the values above 10.\n",
    "\n",
    "**Check the values greater than 10 in the `y` column** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24067</th>\n",
       "      <td>2.00</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI2</td>\n",
       "      <td>58.9</td>\n",
       "      <td>57.0</td>\n",
       "      <td>12210</td>\n",
       "      <td>8.09</td>\n",
       "      <td>58.90</td>\n",
       "      <td>19.73</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25998</th>\n",
       "      <td>4.01</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>I1</td>\n",
       "      <td>61.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>15223</td>\n",
       "      <td>10.14</td>\n",
       "      <td>10.10</td>\n",
       "      <td>6.17</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27415</th>\n",
       "      <td>5.01</td>\n",
       "      <td>Fair</td>\n",
       "      <td>J</td>\n",
       "      <td>I1</td>\n",
       "      <td>65.5</td>\n",
       "      <td>59.0</td>\n",
       "      <td>18018</td>\n",
       "      <td>10.74</td>\n",
       "      <td>10.54</td>\n",
       "      <td>6.97</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27630</th>\n",
       "      <td>4.50</td>\n",
       "      <td>Fair</td>\n",
       "      <td>J</td>\n",
       "      <td>I1</td>\n",
       "      <td>65.8</td>\n",
       "      <td>58.0</td>\n",
       "      <td>18531</td>\n",
       "      <td>10.23</td>\n",
       "      <td>10.16</td>\n",
       "      <td>6.71</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49189</th>\n",
       "      <td>0.51</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>61.8</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2075</td>\n",
       "      <td>5.15</td>\n",
       "      <td>31.80</td>\n",
       "      <td>5.12</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat      cut color clarity  depth  table  price      x      y      z  \\\n",
       "24067   2.00  Premium     H     SI2   58.9   57.0  12210   8.09  58.90  19.73   \n",
       "25998   4.01  Premium     I      I1   61.0   61.0  15223  10.14  10.10   6.17   \n",
       "27415   5.01     Fair     J      I1   65.5   59.0  18018  10.74  10.54   6.97   \n",
       "27630   4.50     Fair     J      I1   65.8   58.0  18531  10.23  10.16   6.71   \n",
       "49189   0.51    Ideal     E     VS1   61.8   55.0   2075   5.15  31.80   5.12   \n",
       "\n",
       "       clarity_rating  color_rating  \n",
       "24067               1             4  \n",
       "25998               0             5  \n",
       "27415               0             6  \n",
       "27630               0             6  \n",
       "49189               4             1  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "diamonds.loc[diamonds['y'] > 10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can clearly see that the 31.8 in row 49189 is an outlier for the y value. Also, we can see that the 58.9 value for `y` in row 24067 is actually its depth, so it was a mistake when they introduced the data. Let's create a function to fix these outliers.\n",
    "\n",
    "**Create a function named `calculate_y` to calculate `y` using `z` and `x` the same way you did above*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n",
    "def calculate_y(row):\n",
    "    return round((2*row['z']/(row['depth']/100))-row['x'], 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "diamonds['y'] = np.where(diamonds['y'] > 10, diamonds.apply(lambda row: calculate_y(row), axis=1) , diamonds['y'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(53932, 12)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "carat                   2\n",
       "cut               Premium\n",
       "color                   H\n",
       "clarity               SI2\n",
       "depth                58.9\n",
       "table                  57\n",
       "price               12210\n",
       "x                    8.09\n",
       "y                    58.9\n",
       "z                   19.73\n",
       "clarity_rating          1\n",
       "color_rating            4\n",
       "Name: 24067, dtype: object"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.loc[24067] # No change, probably something is wrong with more than one measure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "carat              0.51\n",
       "cut               Ideal\n",
       "color                 E\n",
       "clarity             VS1\n",
       "depth              61.8\n",
       "table                55\n",
       "price              2075\n",
       "x                  5.15\n",
       "y                 11.42\n",
       "z                  5.12\n",
       "clarity_rating        4\n",
       "color_rating          1\n",
       "Name: 49189, dtype: object"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.loc[49189]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will check the rows that had an outlier in `y`, to check that the values were changed.\n",
    "\n",
    "**Check those rows (also validating with your function) and comment what you see**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24067</th>\n",
       "      <td>2.00</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI2</td>\n",
       "      <td>58.9</td>\n",
       "      <td>57.0</td>\n",
       "      <td>12210</td>\n",
       "      <td>8.09</td>\n",
       "      <td>58.90</td>\n",
       "      <td>19.73</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25998</th>\n",
       "      <td>4.01</td>\n",
       "      <td>Premium</td>\n",
       "      <td>I</td>\n",
       "      <td>I1</td>\n",
       "      <td>61.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>15223</td>\n",
       "      <td>10.14</td>\n",
       "      <td>10.09</td>\n",
       "      <td>6.17</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27415</th>\n",
       "      <td>5.01</td>\n",
       "      <td>Fair</td>\n",
       "      <td>J</td>\n",
       "      <td>I1</td>\n",
       "      <td>65.5</td>\n",
       "      <td>59.0</td>\n",
       "      <td>18018</td>\n",
       "      <td>10.74</td>\n",
       "      <td>10.54</td>\n",
       "      <td>6.97</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27630</th>\n",
       "      <td>4.50</td>\n",
       "      <td>Fair</td>\n",
       "      <td>J</td>\n",
       "      <td>I1</td>\n",
       "      <td>65.8</td>\n",
       "      <td>58.0</td>\n",
       "      <td>18531</td>\n",
       "      <td>10.23</td>\n",
       "      <td>10.17</td>\n",
       "      <td>6.71</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49189</th>\n",
       "      <td>0.51</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>E</td>\n",
       "      <td>VS1</td>\n",
       "      <td>61.8</td>\n",
       "      <td>55.0</td>\n",
       "      <td>2075</td>\n",
       "      <td>5.15</td>\n",
       "      <td>11.42</td>\n",
       "      <td>5.12</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat      cut color clarity  depth  table  price      x      y      z  \\\n",
       "24067   2.00  Premium     H     SI2   58.9   57.0  12210   8.09  58.90  19.73   \n",
       "25998   4.01  Premium     I      I1   61.0   61.0  15223  10.14  10.09   6.17   \n",
       "27415   5.01     Fair     J      I1   65.5   59.0  18018  10.74  10.54   6.97   \n",
       "27630   4.50     Fair     J      I1   65.8   58.0  18531  10.23  10.17   6.71   \n",
       "49189   0.51    Ideal     E     VS1   61.8   55.0   2075   5.15  11.42   5.12   \n",
       "\n",
       "       clarity_rating  color_rating  \n",
       "24067               1             4  \n",
       "25998               0             5  \n",
       "27415               0             6  \n",
       "27630               0             6  \n",
       "49189               4             1  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.loc[diamonds['y'] > 10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24067</th>\n",
       "      <td>2.0</td>\n",
       "      <td>Premium</td>\n",
       "      <td>H</td>\n",
       "      <td>SI2</td>\n",
       "      <td>58.9</td>\n",
       "      <td>57.0</td>\n",
       "      <td>12210</td>\n",
       "      <td>8.09</td>\n",
       "      <td>58.9</td>\n",
       "      <td>19.73</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat      cut color clarity  depth  table  price     x     y      z  \\\n",
       "24067    2.0  Premium     H     SI2   58.9   57.0  12210  8.09  58.9  19.73   \n",
       "\n",
       "       clarity_rating  color_rating  \n",
       "24067               1             4  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.loc[diamonds['z'] > 10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# There is still one outlier for which calculation of y gave the same result. Probably more than one value is incorrect for\n",
    "# this row so it's better to drop the whole row.\n",
    "diamonds.drop(24067, axis=0, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have corrected or dropped all of our outliers, lets plot another box plot to double check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAEhCAYAAACuva4PAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAWyElEQVR4nO3de2xe9X3H8c/HNhcna9by9GlgCeEiI+gE4uZV7RDRSi41tKJrtUkgdXFvM9KYY9ila6VtVasyrdM2jRip1AKKozKmlhZta4NLYGMUiQIG0iY0VLgsbRMIeXC4tYESx9/9YQclXuLYic/v95zj90uy7N/zHHM+Cs7Jx79z+TkiBAAAgOK15A4AAAAwX1C8AAAAEqF4AQAAJELxAgAASITiBQAAkEgpipftodwZgPmIv3vlxv8/II/p/u61pQxytBYtWvSBzs5OnnsBpPdq7gA4ehw7gWwOe+wsRfE666yzNDw8nDsGMO/YfiZ3Bhw9jp1AHtMdO0txqhEAAKAKKF4AAACJULwAAAASoXgBAAAkQvECAABIhOIFACVg+2zbmw74eNX2dblzAZgdihcKNTo6qrVr12p0dDR3FKDUIuInEXFBRFwg6WJJeyTdnTkWCsBxs9ooXijU4OCgNm/erPXr1+eOAlTJCkk/jYif5Q6Cucdxs9ooXijM6OiohoaGFBEaGhritzdg7lwl6c7cITD3OG5WH8ULhRkcHNT4+Lgkad++ffz2BswB28dLulLSNw/zfo/tYdvDjUYjbTgcM46b1UfxQmHuu+8+jY2NSZLGxsa0cePGzImASrhc0hMR8cKh3oyIgYjojIjOer2eOBqOFcfN6qN4oTArV65UW9vEcqBtbW1atWpV5kRAJVwtTjNWFsfN6qN4oTDd3d1qaZn4EWttbdWaNWsyJwLKzfYCSaskfTt3FhSD42b1UbxQmFqtpq6uLtlWV1eXarVa7khAqUXEnoioRcQrubOgGBw3q68tdwBUW3d3t7Zt28ZvbQAwQxw3q43ihULVajWtW7cudwwAKA2Om9XGqUYAAIBEKF4AADQRlgyqNooXAABNhCWDqo3iBQBAk2DJoOorrHjZvs32LttbDnjtD20/ZXvcdmdR+wYAoIxYMqj6ipzxul1S15TXtkj6qKQHC9wvAAClxJJB1VdY8YqIByXtnvLa1oj4SVH7BACgzFgyqPqa9hov2z22h20PNxqN3HEAACgcSwZVX9MWr4gYiIjOiOis1+u54wAAUDiWDKo+nlwPAEATYcmgaqN4AQDQRFgyqNqKfJzEnZIelnS27e22P2X7I7a3S3qfpO/a/l5R+wcAAGg2Rd7VeHVEnBIRx0XE0oi4NSLunvz6hIhYHBEfKGr/AACUEUsGVVvTXlwPAMB8xJJB1UbxAgCgSbBkUPVRvAAAaBIsGVR9FC8AAJoESwZVH8ULAIAmcemll047RvlRvFAo7s4BgJmLiNwRUDCKFwrF3TkAMHPf//73Dxo/+OCDmZKgKBQvFIa7cwBgdhYvXjztGOVH8UJhuDsHAGZn586d045RfhQvFIa7cwBgdur1+rRjlB/FC4VZuXKl2tom1mFva2vTqlWrMicCgOb23HPPTTtG+VG8UJju7m61tEz8iLW2tmrNmjWZEwEAkBfFC4Wp1Wrq6uqSbXV1dalWq+WOBABNjed4VV9b7gCotu7ubm3bto3ZLgCYgeOPP/6g8QknnJApCYrCjBcKVavVtG7dOma7AGAGeI5X9VG8AABoEjzHq/ooXgAANIkXXnhh2jHKj+KFQrFWIwDM3NTH7qxevTpTEhSF4oVCsVYjMHdsv932Xbaftr3V9vtyZ8LcWr58+bRjlB/FC4U5cK3Ge+65h1kv4NjdKGkoIs6RdL6krZnzYI7ddNNNB437+/szJUFRKF4ozODgoPbu3StJ2rt3L7NewDGwvUjSckm3SlJEvBkRL+dNhbm2bdu2accoP4oXCrNx40ZFhCQpInTvvfdmTgSU2pmSGpK+ZvtJ27fYXjh1I9s9todtDzcajfQpcUwWLlw47RjlR/FCYbgtGphTbZIukvSViLhQ0q8kfXbqRhExEBGdEdHJAsvl8/rrr087RvkVVrxs32Z7l+0tB7x2ku2Ntp+Z/PyOovaP/LgtGphT2yVtj4hHJsd3aaKIoUL2nyU43BjlV+SM1+2Suqa89llJ90fEWZLu1yF+W0N1cFs0MHciYqekX9g+e/KlFZJ+nDESCjB1iSCWDKqewopXRDwoafeUlz8saXDy60FJv1/U/pFfd3f3QWPWawSOWa+kO2z/SNIFkv4ucx7MsTfeeGPaMcov9SLZiyPieUmKiOdtv+twG9rukdQjScuWLUsUD3PppZde+n9j1mwEjl5EbJLUmTsHgKPXtBfXc4Fo+X3pS1+adgwAwHyTuni9YPsUSZr8vCvx/pEQz6MBgNmxPe0Y5Ze6eP2HpP0X/nRL+vfE+0dCS5cunXYMADjY1MsxuDyjeop8nMSdkh6WdLbt7bY/JenvJa2y/YykVZNjVNSpp5560Jhr9QBgei+++OK0Y5RfYRfXR8TVh3lrRVH7RHN57LHHDho/+uijmZIAANAcUt/ViHmEBwECaAb9/f0aGRnJHeOo9fX15Y4wIx0dHert7c0do+k17V2NKL8VKw6e3Fy5cmWmJABQDm1tbdOOUX78H0VhrrnmmrcWyratnp6e3JEAzENlmoUZGRnRpz/96bfGN998szo6OjImwlxjxguFqdVqOvnkkyVJp5xyCnfnAMARdHR0vDXLtXjxYkpXBVG8UJjR0dG3FsbeuXOnRkdHMycCgOZ3xhlnqKWlRTfccEPuKCgAxQuFGRgY0Pj4uCRpfHxcAwMDmRMBQPNbsGCBzjvvPGa7KorihcLcf//9044BAJhvKF4oDI+TAADgYBQvFIbHSQAAcDCKFwqzevXqaccAAMw3FC8U5qabbjpo3N/fnykJAADNgeKFwmzbtm3aMQAA8w3FC4VpaWmZdgwAwHzDv4QozP5neB1uDADAfEPxAgAASITiBQAAkAjFCwAAIJG23AEwc/39/RoZGckdY8ZaWloOuq6rpaVFfX19GRPNTEdHh3p7e3PHAABUEDNeKMxpp5120Pj000/PEwQAgCbBjFeJlHEW5rLLLtP4+LgWLlyo2267LXccAACyYsYLhdo/6/XFL34xcxIAAPKjeKFQixYt0vnnn6+LL744dxQAALKjeAEAACSSpXjZ7rO9xfZTtq/LkQEAACC15MXL9rmS/ljSeySdL+lDts9KnQMAACC1HDNe75b0g4jYExFjkv5H0kcy5AAAAEgqR/HaImm57ZrtBZKukHTq1I1s99getj3caDSShwQAAJhryYtXRGyV9GVJGyUNSfqhpLFDbDcQEZ0R0Vmv1xOnBAAAmHtZLq6PiFsj4qKIWC5pt6RncuQAgDKxvc32ZtubbA/nzgNg9rI8ud72uyJil+1lkj4q6X05cgBACb0/Il7MHQLA0cm1ZNC3bNck7ZV0bUS8lCkHAABAMlmKV0RcmmO/AFByIele2yHpqxExMHUD2z2SeiRp2bJlieMBOBKeXA8A5XFJRFwk6XJJ19pePnUDbkwCmhvFCwBKIiKem/y8S9LdmngQNYASoXgBQAnYXmj7bfu/lrRaE89FBFAiuS6uBwDMzmJJd9uWJo7d/xoRQ3kjAZgtihcAlEBEPKuJ9W0BlBinGgEAABKheAEAACRC8QIAAEiE4gUAAJAIxQsAACARihcAAEAiFC8AAIBEKF4AAACJULwAAAASoXgBAAAkQvECAABIhOIFAACQCMULAAAgEYoXAABAIhQvAACARCheAAAAiVC8AAAAEqF4AQAAJJKleNm+3vZTtrfYvtP2iTlyAAAApJS8eNleImmtpM6IOFdSq6SrUucAAABILdepxjZJ7bbbJC2Q9FymHAAAAMkkL14RsUPSP0r6uaTnJb0SEfdO3c52j+1h28ONRiN1TAAAgDmX41TjOyR9WNIZkn5L0kLbH5u6XUQMRERnRHTW6/XUMQEAAOZcjlONKyX9b0Q0ImKvpG9L+t0MOQAAAJLKUbx+Lum9thfYtqQVkrZmyAEAAJBUjmu8HpF0l6QnJG2ezDCQOgcAAEBqbTl2GhGfl/T5HPsGAADIhSfXAwAAJELxAgAASITiBQAAkAjFCwAAIBGKFwAAQCIULwAAgEQoXgBQIrZbbT9p+zu5swCYvSM+x8v2n0q6IyJeSpAHADC9Pk2s9rEox877+/s1MjKSY9fzxv4/376+vsxJqq+jo0O9vb1J9zmTB6ieLOkx209Iuk3S9yIiio0FANVl+35J/xQRGw54bSAieo7wfUslfVDSDZL+rNiUhzYyMqJNW7Zq34KTcux+Xmh5c+Kf2MeffSFzkmpr3bM7y36PWLwi4q9t/42k1ZI+Iekm29+QdGtE/LTogABQQWdI+ivbvxMRX5h8rXMG3/cvkj4j6W2H28B2j6QeSVq2bNmx5jykfQtO0uvnXFHIfxtIpf3pDUfeqAAzusZrcoZr5+THmKR3SLrL9j8UmA0AquplSSskLbb9n7Z/80jfYPtDknZFxOPTbRcRAxHRGRGd9Xp9juICmCszucZrraRuSS9KukXSX0bEXtstkp7RxG9fAICZc0SMSfoT2x+X9JAmfqGdziWSrrR9haQTJS2y/fWI+FixUQHMpZlc4/VOSR+NiJ8d+GJEjE/+BgYAmJ2b938REbfb3izp2um+ISI+J+lzkmT79yT9BaULKJ+ZXOP1t9O8t3Vu4wBA9UXEV6eMH5f0yUxxACQ0kxkvAEATiYgHJD2QOQaAo8ADVAEAABKheAEAACRC8QIAAEiE4gUAAJAIxQsAACARihcAAEAiFC8AAIBEKF4AAACJJC9ets+2vemAj1dtX5c6BwAAQGrJn1wfET+RdIEk2W6VtEPS3alzAAAApJb7VOMKST+dugA3AABAFeUuXldJuvNQb9jusT1se7jRaCSOBQAAMPeyFS/bx0u6UtI3D/V+RAxERGdEdNbr9bThAAAACpD8Gq8DXC7piYh4IWMGAMAs7NixQ617XlH70xtyRwGOSeueUe3YMZZ8vzlPNV6tw5xmBAAAqKIsM162F0haJemaHPsHABydJUuWaOev2/T6OVfkjgIck/anN2jJksXJ95uleEXEHkm1HPsGAADIJfddjQAAAPMGxQsAACARihcAAEAiFC8AAIBEKF4AAACJULwAAAASoXgBAAAkQvECAABIJOdajVn19/drZGQkd4zK2/9n3NfXlzlJtXV0dKi3tzd3DADAEczb4jUyMqJNW7Zq34KTckeptJY3Q5L0+LOshV6U1j27c0cAAMzQvC1ekrRvwUmsN4bSa396Q+4IAIAZ4hovAACARCheAAAAiVC8AAAAEqF4AQAAJELxAgAASITiBQAlYPtE24/a/qHtp2x/IXcmALM3rx8nAQAl8mtJl0XEL20fJ+kh2/dExA9yBwMwcxQvACiBiAhJv5wcHjf5EfkSATganGoEgJKw3Wp7k6RdkjZGxCOH2KbH9rDt4UajkT4kgGlRvACgJCJiX0RcIGmppPfYPvcQ2wxERGdEdNbr9fQhAUyL4gUAJRMRL0t6QFJX5igAZolrvACgBGzXJe2NiJdtt0taKenLObK07tnNGqEFannjVUnS+ImLMiepttY9uyUtTr7fLMXL9tsl3SLpXE1cHPrJiHg4RxYAKIlTJA3abtXE2YpvRMR3Uofo6OhIvct5Z2TkNUlSx5npS8H8sjjLz3OuGa8bJQ1FxB/YPl7Sgkw5AKAUIuJHki7MnaO3tzd3hMrr6+uTJN14442Zk6AIyYuX7UWSlkv6uCRFxJuS3kydAwAAILUcF9efKakh6Wu2n7R9i+2FUzfilmgAAFA1OYpXm6SLJH0lIi6U9CtJn526EbdEAwCAqslRvLZL2n7Ag//u0kQRAwAAqLTkxSsidkr6he2zJ19aIenHqXMAAACkluuuxl5Jd0ze0fispE9kygEAAJBMluIVEZskdebYNwAAQC4sGQQAAJAIxQsAACCRebtW444dO9S65xXWG0Ppte4Z1Y4dY7ljAABmgBkvAACARObtjNeSJUu089dtev2cK3JHAY5J+9MbtGQJi+kCQBkw4wUAAJAIxQsAACARihcAAEAiFC8AAIBEKF4AAACJULwAAAASoXgBAAAkQvECAABIhOIFAACQCMULAAAgEYoXAABAIhQvAACARCheAAAAiVC8AAAAEqF4AQAAJELxAgAASITiBQAAkAjFCwAAIJG2HDu1vU3Sa5L2SRqLiM4cOQAAAFLKUrwmvT8iXsy4fwAAgKQ41QgAAJBIruIVku61/bjtnkNtYLvH9rDt4UajkTgeADQX26fa/m/bW20/ZbsvdyYAs5ereF0SERdJulzStbaXT90gIgYiojMiOuv1evqEANBcxiT9eUS8W9J7NXHs/O3MmQDMUpbiFRHPTX7eJeluSe/JkQMAyiIino+IJya/fk3SVklL8qYCMFvJi5fthbbftv9rSaslbUmdAwDKyvbpki6U9Mgh3uMyDaCJ5ZjxWizpIds/lPSopO9GxFCGHABQOrZ/Q9K3JF0XEa9OfZ/LNIDmlvxxEhHxrKTzU+8XAMrO9nGaKF13RMS3c+cBMHs8TgIASsC2Jd0qaWtE/HPuPACODsULAMrhEkl/JOky25smP67IHQrA7OR8cn12rXt2q/3pDbljVFrLGxOXoIyfuChzkupq3bNbE5dOosoi4iFJzp0DwLGZt8Wro6Mjd4R5YWTkNUlSx5kUg+Is5ucZAEpi3hav3t7e3BHmhb6+iYdr33jjjZmTAACQH9d4AQAAJELxAgAASITiBQAAkAjFCwAAIBGKFwAAQCIULwAAgEQoXgAAAIlQvAAAABKheAEAACRC8QIAAEiE4gUAAJAIxQsAACARihcAAEAiFC8AAIBEKF4AAACJULwAAAASacsdAACAIvX392tkZCR3jBnbn7Wvry9zktnp6OhQb29v7hhNL9uMl+1W20/a/k6uDAAANJv29na1t7fnjoGC5Jzx6pO0VdKijBkAABXHLAyaSZYZL9tLJX1Q0i059g8AAJBDrlON/yLpM5LGM+0fAAAgueTFy/aHJO2KiMePsF2P7WHbw41GI1E6AACA4uSY8bpE0pW2t0n6N0mX2f761I0iYiAiOiOis16vp84IAAAw55IXr4j4XEQsjYjTJV0l6b8i4mOpcwAAAKTGA1QBAAASyfoA1Yh4QNIDOTMAAACkwowXAABAIhQvAACayOjoqNauXavR0dHcUVAAihcAAE1kcHBQmzdv1vr163NHQQEoXgAANInR0VENDQ0pIjQ0NMSsVwVlvbges9Pf3//WqvVlsT9vX19f5iQz19HRwdpuALIYHBzU+PjEoi779u3T+vXrdf3112dOhbnEjBcK1d7ervb29twxgNKzfZvtXba35M6C4tx3330aGxuTJI2NjWnjxo2ZE2GuMeNVIszCAPPa7ZJuksSFPxW2cuVKbdiwQWNjY2pra9OqVatyR8IcY8YLAEogIh6UtDt3DhSru7tbLS0T/zS3trZqzZo1mRNhrlG8AKBCbPfYHrY93Gg0csfBLNVqNXV1dcm2urq6VKvVckfCHKN4AUCFRMRARHRGRGe9Xs8dB0ehu7tb5513HrNdFcU1XgAANJFaraZ169bljoGCMOMFAACQCMULAErA9p2SHpZ0tu3ttj+VOxOA2eNUIwCUQERcnTsDgGPHjBcAAEAiFC8AAIBEKF4AAACJOCJyZzgi2w1JP8udA0ftnZJezB0CR+W0iOBhUCXFsbPUOG6W22GPnaUoXig328MR0Zk7BwCUBcfN6uJUIwAAQCIULwAAgEQoXkhhIHcAACgZjpsVxTVeAAAAiTDjBQAAkAjFCwAAIBGKFwAAQCIULwAAgEQoXgAAAIn8Hwa9l8mMuSuhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Run this code\n",
    "fig, ax = plt.subplots(1,2, figsize=(10, 5))\n",
    "sns.boxplot(y=diamonds.y, ax=ax[0])\n",
    "sns.boxplot(y=diamonds.z, ax=ax[1])\n",
    "plt.subplots_adjust(wspace=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What do you think? Are these values more reasonable?**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nWe have much more outliers now but the range of this values is moch more reasonable. There is though one oultier in z which\\nvalue is around 1, I think this should be checked and recalculated as well.\\n'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your thoughts here\n",
    "'''\n",
    "We have much more outliers now but the range of this values is moch more reasonable. There is though one oultier in z which\n",
    "value is around 1, I think this should be checked and recalculated as well.\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>carat</th>\n",
       "      <th>cut</th>\n",
       "      <th>color</th>\n",
       "      <th>clarity</th>\n",
       "      <th>depth</th>\n",
       "      <th>table</th>\n",
       "      <th>price</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "      <th>z</th>\n",
       "      <th>clarity_rating</th>\n",
       "      <th>color_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14635</th>\n",
       "      <td>1.07</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>F</td>\n",
       "      <td>SI1</td>\n",
       "      <td>60.6</td>\n",
       "      <td>57.0</td>\n",
       "      <td>5909</td>\n",
       "      <td>6.62</td>\n",
       "      <td>6.67</td>\n",
       "      <td>1.07</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20694</th>\n",
       "      <td>1.53</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>I</td>\n",
       "      <td>SI1</td>\n",
       "      <td>61.9</td>\n",
       "      <td>54.0</td>\n",
       "      <td>8971</td>\n",
       "      <td>7.43</td>\n",
       "      <td>7.50</td>\n",
       "      <td>1.53</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21654</th>\n",
       "      <td>1.41</td>\n",
       "      <td>Ideal</td>\n",
       "      <td>H</td>\n",
       "      <td>VS1</td>\n",
       "      <td>60.7</td>\n",
       "      <td>56.0</td>\n",
       "      <td>9752</td>\n",
       "      <td>7.31</td>\n",
       "      <td>7.22</td>\n",
       "      <td>1.41</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       carat    cut color clarity  depth  table  price     x     y     z  \\\n",
       "14635   1.07  Ideal     F     SI1   60.6   57.0   5909  6.62  6.67  1.07   \n",
       "20694   1.53  Ideal     I     SI1   61.9   54.0   8971  7.43  7.50  1.53   \n",
       "21654   1.41  Ideal     H     VS1   60.7   56.0   9752  7.31  7.22  1.41   \n",
       "\n",
       "       clarity_rating  color_rating  \n",
       "14635               2             2  \n",
       "20694               2             5  \n",
       "21654               4             4  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.loc[diamonds['z'] < 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "diamonds['z'] = np.where(diamonds['z'] < 2, diamonds.apply(lambda row: calculate_z(row), axis=1), diamonds['z'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "carat              1.07\n",
       "cut               Ideal\n",
       "color                 F\n",
       "clarity             SI1\n",
       "depth              60.6\n",
       "table                57\n",
       "price              5909\n",
       "x                  6.62\n",
       "y                  6.67\n",
       "z                  4.03\n",
       "clarity_rating        2\n",
       "color_rating          2\n",
       "Name: 14635, dtype: object"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diamonds.loc[14635]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAEhCAYAAACuva4PAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVkUlEQVR4nO3df2ydV33H8c/HNogkI6PcXpI2aZoyV2USXUp7Va0rjUR+4Zaq/NAmtRLDbDAjrXPcbRoDaRsaomibpok2SCtWW2oEFEFptI21XtOyrlSCtk4JTUqCajoDCU1z4xYKCyNx/N0fcbrYOI6v6+ccP4/fL8mKz71XnI8gnHzuuc89jyNCAAAAKF5b7gAAAACLBcULAAAgEYoXAABAIhQvAACARCheAAAAiZSieNkezJ0BWIz4/x4AzK+O3AFmY/ny5W9vNBqcewGk91LuAJi7s88+O9auXZs7BrDo7Ny583BE1Kd7rhTF68ILL9TQ0FDuGMCiY/uZ3Bkwd2vXrmXtBDKw/YPTPVeKjxoBAACqgOIFAACQCMULAAAgEYoXAABAIhQvAACARCheAFACti+yveuUn5ds35Q7F4DWULxQqNHRUW3dulWjo6O5owClFhHfi4hLIuISSZdJOiJpe+ZYKADrZrVRvFCogYEB7d69W5/73OdyRwGqZKOk70fEac8KQnmxblYbxQuFGR0d1eDgoCJCg4ODvHsD5s/1ku6e7gnbPbaHbA81m83EsfBKsW5WH8ULhRkYGND4+Lgk6fjx47x7A+aB7VdLuk7SV6Z7PiL6I6IREY16fdo7lmABY92sPooXCvPggw9qbGxMkjQ2NqYdO3ZkTgRUwtWSnoyI53MHwfxj3aw+ihcKs2nTJnV0nLgdaEdHhzZv3pw5EVAJN+g0HzOi/Fg3q4/ihcJ0d3erre3EX7H29na9733vy5wIKDfbSyVtlnRv7iwoButm9VG8UJharaauri7ZVldXl2q1Wu5IQKlFxJGIqEXET3NnQTFYN6uvI3cAVFt3d7dGRkZ41wYAs8S6WW0ULxSqVqvp1ltvzR0DAEqDdbPa+KgRAAAgEYoXAABAIhQvAACARCheAAAsINwku9oKK16277R9yPaeUx77PdtP2x633ShqbgAAyoqbZFdbkTted0nqmvLYHknvkfRIgfMCAFBK3CS7+gorXhHxiKQXpjy2NyK+V9ScAACUGTfJrr4Fe42X7R7bQ7aHms1m7jgAABSOm2RX34ItXhHRHxGNiGjU6/XccQAAKBw3ya6+BVu8AABYbLhJdvVRvAAAWCC4SXb1FXmcxN2SvinpItv7bX/A9rtt75d0haR/t/0fRc0PAEAZdXd36+KLL2a3q6IKu0l2RNxwmqe2FzUnAABlx02yq42PGgEAABKheAEAACRC8QIAAEiE4gUAAJAIxQsAgAVkdHRUW7du5T6NFUXxQqFYQACgNQMDA9q9ezf3aawoihcKxQICALM3OjqqwcFBRYQGBwd501pBFC8UhgUEAFozMDCg8fFxSdLx48d501pBFC8UhgUEAFrz4IMPamxsTJI0NjamHTt2ZE6E+UbxQmFYQACgNVddddWMY5QfxQuF2bRpkzo6TtyVqqOjQ5s3b86cCAAWtojIHQEFo3ihMN3d3WprO/FXrL29nRu+AsAZPProo5PG3/jGNzIlQVEoXihMrVZTV1eXbKurq0u1Wi13JABY0N761rdOGvNRY/V05A6Aauvu7tbIyAi7XQAwC7ZzR0DB2PFCoWq1mm699VZ2uwBgFh555JEZxyg/ihcAAAvEihUrZhyj/CheAAAsEAcPHpxxjPKjeKFQ3KsRAGZv5cqVM45RfhQvFIp7NQLA7D3//PMzjlF+FC8U5tR7Nd5///3segHAGaxfv37GMcqP4oXCDAwM6NixY5KkY8eOsesFAGfAyfXVR/FCYXbs2PHyIhIReuCBBzInAoCFbepJ9RwnUT0ULxSGr0UDQGtYN6uvsOJl+07bh2zvOeWx19veYfuZiT/PKmp+5MdFogDQGtbN6ityx+suSV1THvuIpIci4kJJD02MUVGbN2+eNN6yZUumJABQDqyb1VdY8YqIRyS9MOXhd0oamPh9QNK7ipof+XV3d08ac79GAJgZ62b1pb7Ga0VEPCdJE3++4XQvtN1je8j2ULPZTBYQ8+fFF1+ccQygNbZfZ/se2/ts77V9Re5MmF+sm9W3YC+uj4j+iGhERKNer+eOgzn4xCc+MeMYQMtukTQYEW+StE7S3sx5MM9YN6svdfF63vY5kjTx56HE8yOhkZGRGccAZs/2cknrJd0hSRFxNCJ+kjcV5hvrZvWlLl7/KunkB9jdkv4l8fxIaPXq1TOOAbTkjZKakj5r+9u2b7e9LHcozK9zzjln0vjcc8/NlARFKfI4ibslfVPSRbb32/6ApL+TtNn2M5I2T4xRUeedd96k8Zo1azIlASqhQ9Klkv45It4i6X80zTfDuT62WjjJvno6ivoPjogbTvPUxqLmxMLyxBNPTBo//vjjmZIAlbBf0v6IeGxifI+mKV4R0S+pX5IajQb/apfMc889N+MY5bdgL65H+U19p8Y7N2DuIuKgpB/ZvmjioY2SvpsxEgow9aPGqWOUH8ULhdm4cfLm5qZNmzIlASqjV9IXbD8l6RJJn8ycB/Ps6NGjk8bHjh3LlARFoXihMB/60IdkW5JkWz09PZkTAeUWEbsmjtn5rYh4V0RwyFPFjI6OThofPnw4UxIUheKFwtRqNa1cuVLSie3yWq2WOREAAHlRvFCY0dHRl2/wevDgwV95JwcAmKy9vX3GMcqP4oXC9Pf3a3x8XJI0Pj6u/v7+zIkAYGE7fvz4jGOUH8ULhXnooYdmHAMAJpt6/uHUMcqP4oXCcJwEALTmrLPOmnGM8qN4oTAcJwEArXnqqadmHKP8KF4ozJYtW2YcAwCw2BR2yyDg05/+9KTxtm3bdNddd+UJA2DR2rZtm4aHh3PHmLO+vr7cEWals7NTvb29uWMseOx4oTAjIyMzjgEAk3V0dMw4RvnxvygK09bW9vJxEifHAJBamXZhhoeH9cEPfvDl8W233abOzs6MiTDf+JcQhTm1dE03BgBM1tnZ+fIu14oVKyhdFUTxAgBgAbngggvU1tamm2++OXcUFIDiBQDAArJ06VJdfPHF7HZVFMULAAAgES6uL5GyfSV6uovry/C1aL4SDQAoCjteKMz5558/abx27do8QQAAWCDY8SqRMu7CbNiwQePj41q2bJnuvPPO3HEAAMiKHS8U6uSu18c//vHMSQAAyI/ihUItX75c69at02WXXZY7CgAA2VG8AAAAEslSvGz32d5j+2nbN+XIAAAAkFry4mX7zZL+SNLlktZJutb2halzAAAApJZjx+s3JX0rIo5ExJik/5L07gw5AAAAkspRvPZIWm+7ZnuppGsknTf1RbZ7bA/ZHmo2m8lDAgAAzLfkxSsi9kr6e0k7JA1K+o6ksWle1x8RjYho1Ov1xCkBAADmX5aL6yPijoi4NCLWS3pB0jM5cgAAAKSU5eR622+IiEO210h6j6QrcuQAAABIKdctg75quybpmKQbI+LFTDkAAACSyVK8IuKqHPMCAADkxMn1AAAAiVC8AAAAEqF4AQAAJELxAgAASITiBQAAkAjFCwAAIBGKFwAAQCIULwAAgEQoXgAAAIlQvAAAABKheAEAACRC8QIAAEiE4gUAAJBIR+4AAIDZsT0i6WeSjksai4hG3kQAWkXxAoByeVtEHM4dAsDc8FEjAABAIhQvACiPkPSA7Z22e6Z7ge0e20O2h5rNZuJ4AM6E4gUA5XFlRFwq6WpJN9peP/UFEdEfEY2IaNTr9fQJAcyI4gUAJRERP57485Ck7ZIuz5sIQKsoXgBQAraX2X7tyd8lbZG0J28qAK3iW40AUA4rJG23LZ1Yu78YEYN5IwFoFcULAEogIp6VtC53DgCvDB81AgAAJJKleNn+U9tP295j+27br8mRAwAAIKXkxcv2KklbJTUi4s2S2iVdnzoHAABAark+auyQtMR2h6Slkn6cKQcAAEAyyYtXRByQ9I+SfijpOUk/jYgHpr6O05cBAEDV5Pio8SxJ75R0gaRzJS2z/d6pr+P0ZQAAUDU5PmrcJOm/I6IZEcck3SvpdzLkAAAASCpH8fqhpN+2vdQnTgLcKGlvhhwAAABJ5bjG6zFJ90h6UtLuiQz9qXMAAACkluXk+oj4mKSP5ZgbAAAgF06uBwAASITiBQAAkAjFCwAAIBGKFwAAQCIULwAAgEQoXgAAAIlQvAAAABI5Y/Gy/ScT91cEAADAKzCbHa+Vkp6w/WXbXRO3+QEAzJHth2xfM+Ux7uABLAJnLF4R8VeSLpR0h6T3S3rG9idt/0bB2QCgqi6Q9Je2T72DRyNXGADpzOoar4gISQcnfsYknSXpHtv/UGA2AKiqn0jaKGmF7X+z/eu5AwFI44z3arS9VVK3pMOSbpf0FxFxzHabpGckfbjYiABQOY6IMUl/bPv9kh7ViTe0ACpuNjfJPlvSeyLiB6c+GBHjtq8tJhYAVNptJ3+JiLts75Z0Y8Y8ABI5Y/GKiL+Z4bm98xsHAKovIj4zZbxT0h9migMgIc7xAgAASITiBQAAkAjFCwAAIBGKFwAAQCIULwAAgEQoXgAAAIlQvAAAABKheAEAACRC8QIAAEgkefGyfZHtXaf8vGT7ptQ5AAAAUpvNvRrnVUR8T9IlkmS7XdIBSdtT5wAAAEgt90eNGyV9f+oNuAEAAKood/G6XtLd0z1hu8f2kO2hZrOZOBYAAMD8y1a8bL9a0nWSvjLd8xHRHxGNiGjU6/W04QAAAAqQc8fraklPRsTzGTMAAAAkk7N43aDTfMwIAABQRVmKl+2lkjZLujfH/AAAADkkP05CkiLiiKRajrkBoMwmjuEZknQgIq5NPf+2bds0PDycetpF5eR/v319fZmTVF9nZ6d6e3uTzpmleAEA5qxP0l5Jy3NMPjw8rF179ur40tfnmH5RaDsakqSdz3IJdJHaj7yQZV6KFwCUhO3Vkt4h6WZJf5Yrx/Glr9cv3nRNrumBebFk331Z5s19jhcAYPY+JenDksZP9wLOQAQWNooXAJSA7WslHYqInTO9jjMQgYWN4gUA5XClpOtsj0j6kqQNtj+fNxKAVlG8AKAEIuKjEbE6ItbqxO3Wvh4R780cC0CLKF4AAACJ8K1GACiZiHhY0sOZYwCYg0VbvDgEMA0OAkwjxyGAAIDWLdrixSGAaXAQYPFyHQIIAGjdoi1eEocAohpyHQIIAGgdF9cDAAAkQvECAABIhOIFAACQCMULAAAgEYoXAABAIhQvAACARCheAAAAiVC8AAAAEqF4AQAAJELxAgAASITiBQAAkAjFCwAAIJEsxcv262zfY3uf7b22r8iRAwAAIKWOTPPeImkwIn7X9qslLc2UAwAAIJnkxcv2cknrJb1fkiLiqKSjqXMAAFp34MABtR/5qZbsuy93FOAVaT8yqgMHxpLPm+OjxjdKakr6rO1v277d9rKpL7LdY3vI9lCz2UyfEgAAYJ7l+KixQ9Klknoj4jHbt0j6iKS/PvVFEdEvqV+SGo1GJE8JAPgVq1at0sFfdugXb7omdxTgFVmy7z6tWrUi+bw5drz2S9ofEY9NjO/RiSIGAABQacmLV0QclPQj2xdNPLRR0ndT5wAAAEgt17caeyV9YeIbjc9K+oNMOQAAAJLJUrwiYpekRo65AQAAcuHkegAAgEQoXgAAAInkusYrOw4BRFXkOgQQANA6drwAAAASWbQ7XhwCiKrIdQggAKB17HgBAAAkQvECAABIhOIFAACQCMULAAAgEYoXAABAIhQvAACARCheAAAAiVC8AAAAEqF4AQAAJELxAgAASITiBQAAkAjFCwAAIBGKFwCUgO3X2H7c9ndsP237b3NnAtC6jtwBAACz8ktJGyLi57ZfJelR2/dHxLdyBwMwexQvACiBiAhJP58YvmriJ/IlAjAXfNQIACVhu932LkmHJO2IiMdyZwLQGooXAJRERByPiEskrZZ0ue03T32N7R7bQ7aHms1m+pAAZkTxAoCSiYifSHpYUtc0z/VHRCMiGvV6PXk2ADPLUrxsj9jebXuX7aEcGQCgTGzXbb9u4vclkjZJ2pc3FYBW5by4/m0RcTjj/ABQJudIGrDdrhNvmr8cEV/LEaT9yAtasu++HFMvCm3/+5Ikafw1yzMnqbb2Iy9IWpF8Xr7VCAAlEBFPSXpL7hydnZ25I1Te8PDPJEmdb0xfChaXFVn+PucqXiHpAdsh6TMR0T/1BbZ7JPVI0po1axLHAwBMp7e3N3eEyuvr65Mk3XLLLZmToAi5Lq6/MiIulXS1pBttr5/6Ai4QBQAAVZOleEXEjyf+PCRpu6TLc+QAAABIKXnxsr3M9mtP/i5pi6Q9qXMAAACkluMarxWStts+Of8XI2IwQw4AAICkkheviHhW0rrU8wIAAOTGyfUAAACJULwAAAASWdQHqHL6cvE4gbl4uU5fBgC0btEWL05fToMTmFPIc/oyAKB1i7Z4cfpyGpzADADA/+MaLwAAgEQoXgAAAIlQvAAAABKheAEAACRC8QIAAEiE4gUAAJAIxQsAACARihcAAEAiFC8AAIBEKF4AAACJULwAAAASoXgBAAAkQvECAABIhOIFAACQCMULAAAgEYoXAABAIhQvAACARLIVL9vttr9t+2u5MgAAAKSUc8erT9LejPMDAAAklaV42V4t6R2Sbs8xPwAAQA65drw+JenDksYzzQ8AAJBc8uJl+1pJhyJi5xle12N7yPZQs9lMlA4AAKA4OXa8rpR0ne0RSV+StMH256e+KCL6I6IREY16vZ46IwAAwLxLXrwi4qMRsToi1kq6XtLXI+K9qXMAAACkxjleAAAAiXTknDwiHpb0cM4MAAAAqbDjBQAAkAjFCwAAIBGKFwAAQCIULwAAgESyXlyP1mzbtk3Dw8O5Y7TkZN6+vr7MSWavs7NTvb29uWMAmCdlWzvLuG5KrJ2zxY4XCrVkyRItWbIkdwyg9GyfZ/s/be+1/bTtcv2rjFlj3aw2drxKhHcSwKI2JunPI+JJ26+VtNP2joj4bu5gCx1rJxYSdrwAoAQi4rmIeHLi959J2itpVd5UAFpF8QKAkrG9VtJbJD02zXM9todsDzWbzdTRAJwBxQsASsT2r0n6qqSbIuKlqc9HRH9ENCKiUa/X0wcEMCOKFwCUhO1X6UTp+kJE3Js7D4DWUbwAoARsW9IdkvZGxD/lzgNgbiheAFAOV0r6fUkbbO+a+LkmdygAreE4CQAogYh4VJJz5wDwyrDjBQAAkAjFCwAAIBGKFwAAQCKOiNwZzsh2U9IPcufAnJ0t6XDuEJiT8yOCw6BKirWz1Fg3y+20a2cpihfKzfZQRDRy5wCAsmDdrC4+agQAAEiE4gUAAJAIxQsp9OcOAAAlw7pZUVzjBQAAkAg7XgAAAIlQvAAAABKheAEAACRC8QIAAEiE4gUAAJDI/wGZES/KmUs5EQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Run this code\n",
    "fig, ax = plt.subplots(1,2, figsize=(10, 5))\n",
    "sns.boxplot(y=diamonds.y, ax=ax[0])\n",
    "sns.boxplot(y=diamonds.z, ax=ax[1])\n",
    "plt.subplots_adjust(wspace=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Once you are happy with your cleaning, save the cleaned data and continue to csv. Your new csv should be named ``diamonds_clean``**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n",
    "diamonds.to_csv('diamonds_clean.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}