Working_With_Data.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "---\n",
    "title: Working With Data\n",
    "permalink: /notes/working_with_data\n",
    "key: notes-working-with-data\n",
    "layout: article\n",
    "nav_key: Notes\n",
    "sidebar:\n",
    "  nav: Notes\n",
    "license: false\n",
    "aside:\n",
    "  toc: true\n",
    "show_edit_on_github: false\n",
    "show_date: false\n",
    "---\n",
    "</pre>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with Data\n",
    "\n",
    "By the end of this lesson, you should be able to:\n",
    "- Create a **Panda's DataFrame** and selecting data from DataFrame\n",
    "- Using library to read `CSV` or `EXCEL` file\n",
    "- Give example of **linear regression and classification**\n",
    "- **Split** data randomly into training set and testing set\n",
    "- **Normalize** data using min-max normalization\n",
    "\n",
    "Important Words:\n",
    "- Data Frame\n",
    "- Series\n",
    "- index\n",
    "- column\n",
    "- vectorized operation\n",
    "- normalization\n",
    "- z-norm\n",
    "- min-max\n",
    "- dataset\n",
    "- test, training, validation set"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Short Introduction to Machine Learning\n",
    "\n",
    "In the previous weeks we have discussed how various computation can be done. We begin by discussing computational complexity and divide and conquer strategy through recursion. We then discusses how data can have computation associated with it through object oriented programming. We discussed several data structures and algorithms associated with various problems. We then end up with offering a different perspective by looking at computation as a state machine.\n",
    "\n",
    "In this second half of the course, we will look into how computation can learn from data in order to make a new computation. This new computation is often called a **prediction**. In these lessons, we focus on what is called as **supervised machine learning**. The word supervised machine learning indicates that the computer learns from some existing data on how to compute the prediction. An example of this would be given some images labelled as \"cat\" and \"not a cat\", the computer can learn to predict (or to compute) whether any a new image given to it is a cat or not a cat. \n",
    "\n",
    "cat             |  not a cat | \n",
    ":-------------------------:|:-------------------------:\n",
    "![](https://upload.wikimedia.org/wikipedia/commons/f/ff/Cat_on_laptop_-_Just_Browsing.jpg)  |  ![](https://upload.wikimedia.org/wikipedia/commons/4/49/Catfish.jpeg)\n",
    "\n",
    "Another example would be given a some data of housing prices in Singapore with the year of sale, area, number of rooms, and its floor hight, the computer can predict the price of another house. One other example would be given data of breast cancer cell and its measurements, one can predict whether the cell is malignant or benight. Supervised machine learning assumes that we have some existing data **labelled** with this category \"malignant\" and \"benign\". Using this labelled data (supervised), the computer can predict the category given some new data."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading Data\n",
    "\n",
    "The first step in machine learning would be to understand the data itself. In order to do that we need to be able to read data from some source. One common source is a text file in the form of CSV format (comma separated value). Another common format is Excel spreadsheet. The data can be from some databases or some server. Different data sources will require different ways of handling it. But in all those cases we will need to know how to read those data. \n",
    "\n",
    "For this purpose we will use [Pandas](https://pandas.pydata.org/) library to read our data. We import the data into our Python code by typing the following code.\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "```\n",
    "\n",
    "Now we can use Pandas functions to read the data. For example, if we want to read a CSV file, we can simply type:\n",
    "```python\n",
    "df = pd.read_csv('mydata.csv')\n",
    "```\n",
    "\n",
    "Let's take an example of Singapore housing prices. We can get some of these data from [Data.gov.sg](https://data.gov.sg/). We have downloaded the CSV file so that you can access it simply from the following [dropbox link](https://www.dropbox.com/s/jz8ck0obu9u1rng/resale-flat-prices-based-on-registration-date-from-jan-2017-onwards.csv?raw=1). We can use the url to the raw file to open the CSV in our Python code.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>month</th>\n",
       "      <th>town</th>\n",
       "      <th>flat_type</th>\n",
       "      <th>block</th>\n",
       "      <th>street_name</th>\n",
       "      <th>storey_range</th>\n",
       "      <th>floor_area_sqm</th>\n",
       "      <th>flat_model</th>\n",
       "      <th>lease_commence_date</th>\n",
       "      <th>remaining_lease</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>2 ROOM</td>\n",
       "      <td>406</td>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "      <td>10 TO 12</td>\n",
       "      <td>44.0</td>\n",
       "      <td>Improved</td>\n",
       "      <td>1979</td>\n",
       "      <td>61 years 04 months</td>\n",
       "      <td>232000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>108</td>\n",
       "      <td>ANG MO KIO AVE 4</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1978</td>\n",
       "      <td>60 years 07 months</td>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>602</td>\n",
       "      <td>ANG MO KIO AVE 5</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1980</td>\n",
       "      <td>62 years 05 months</td>\n",
       "      <td>262000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>465</td>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "      <td>04 TO 06</td>\n",
       "      <td>68.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1980</td>\n",
       "      <td>62 years 01 month</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>601</td>\n",
       "      <td>ANG MO KIO AVE 5</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1980</td>\n",
       "      <td>62 years 05 months</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95853</th>\n",
       "      <td>2021-04</td>\n",
       "      <td>YISHUN</td>\n",
       "      <td>EXECUTIVE</td>\n",
       "      <td>326</td>\n",
       "      <td>YISHUN RING RD</td>\n",
       "      <td>10 TO 12</td>\n",
       "      <td>146.0</td>\n",
       "      <td>Maisonette</td>\n",
       "      <td>1988</td>\n",
       "      <td>66 years 04 months</td>\n",
       "      <td>650000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95854</th>\n",
       "      <td>2021-04</td>\n",
       "      <td>YISHUN</td>\n",
       "      <td>EXECUTIVE</td>\n",
       "      <td>360</td>\n",
       "      <td>YISHUN RING RD</td>\n",
       "      <td>04 TO 06</td>\n",
       "      <td>146.0</td>\n",
       "      <td>Maisonette</td>\n",
       "      <td>1988</td>\n",
       "      <td>66 years 04 months</td>\n",
       "      <td>645000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95855</th>\n",
       "      <td>2021-04</td>\n",
       "      <td>YISHUN</td>\n",
       "      <td>EXECUTIVE</td>\n",
       "      <td>326</td>\n",
       "      <td>YISHUN RING RD</td>\n",
       "      <td>10 TO 12</td>\n",
       "      <td>146.0</td>\n",
       "      <td>Maisonette</td>\n",
       "      <td>1988</td>\n",
       "      <td>66 years 04 months</td>\n",
       "      <td>585000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95856</th>\n",
       "      <td>2021-04</td>\n",
       "      <td>YISHUN</td>\n",
       "      <td>EXECUTIVE</td>\n",
       "      <td>355</td>\n",
       "      <td>YISHUN RING RD</td>\n",
       "      <td>10 TO 12</td>\n",
       "      <td>146.0</td>\n",
       "      <td>Maisonette</td>\n",
       "      <td>1988</td>\n",
       "      <td>66 years 08 months</td>\n",
       "      <td>675000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95857</th>\n",
       "      <td>2021-04</td>\n",
       "      <td>YISHUN</td>\n",
       "      <td>EXECUTIVE</td>\n",
       "      <td>277</td>\n",
       "      <td>YISHUN ST 22</td>\n",
       "      <td>04 TO 06</td>\n",
       "      <td>146.0</td>\n",
       "      <td>Maisonette</td>\n",
       "      <td>1985</td>\n",
       "      <td>63 years 05 months</td>\n",
       "      <td>625000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>95858 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         month        town  flat_type block        street_name storey_range  \\\n",
       "0      2017-01  ANG MO KIO     2 ROOM   406  ANG MO KIO AVE 10     10 TO 12   \n",
       "1      2017-01  ANG MO KIO     3 ROOM   108   ANG MO KIO AVE 4     01 TO 03   \n",
       "2      2017-01  ANG MO KIO     3 ROOM   602   ANG MO KIO AVE 5     01 TO 03   \n",
       "3      2017-01  ANG MO KIO     3 ROOM   465  ANG MO KIO AVE 10     04 TO 06   \n",
       "4      2017-01  ANG MO KIO     3 ROOM   601   ANG MO KIO AVE 5     01 TO 03   \n",
       "...        ...         ...        ...   ...                ...          ...   \n",
       "95853  2021-04      YISHUN  EXECUTIVE   326     YISHUN RING RD     10 TO 12   \n",
       "95854  2021-04      YISHUN  EXECUTIVE   360     YISHUN RING RD     04 TO 06   \n",
       "95855  2021-04      YISHUN  EXECUTIVE   326     YISHUN RING RD     10 TO 12   \n",
       "95856  2021-04      YISHUN  EXECUTIVE   355     YISHUN RING RD     10 TO 12   \n",
       "95857  2021-04      YISHUN  EXECUTIVE   277       YISHUN ST 22     04 TO 06   \n",
       "\n",
       "       floor_area_sqm      flat_model  lease_commence_date  \\\n",
       "0                44.0        Improved                 1979   \n",
       "1                67.0  New Generation                 1978   \n",
       "2                67.0  New Generation                 1980   \n",
       "3                68.0  New Generation                 1980   \n",
       "4                67.0  New Generation                 1980   \n",
       "...               ...             ...                  ...   \n",
       "95853           146.0      Maisonette                 1988   \n",
       "95854           146.0      Maisonette                 1988   \n",
       "95855           146.0      Maisonette                 1988   \n",
       "95856           146.0      Maisonette                 1988   \n",
       "95857           146.0      Maisonette                 1985   \n",
       "\n",
       "          remaining_lease  resale_price  \n",
       "0      61 years 04 months      232000.0  \n",
       "1      60 years 07 months      250000.0  \n",
       "2      62 years 05 months      262000.0  \n",
       "3       62 years 01 month      265000.0  \n",
       "4      62 years 05 months      265000.0  \n",
       "...                   ...           ...  \n",
       "95853  66 years 04 months      650000.0  \n",
       "95854  66 years 04 months      645000.0  \n",
       "95855  66 years 04 months      585000.0  \n",
       "95856  66 years 08 months      675000.0  \n",
       "95857  63 years 05 months      625000.0  \n",
       "\n",
       "[95858 rows x 11 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "file_url = 'https://www.dropbox.com/s/jz8ck0obu9u1rng/resale-flat-prices-based-on-registration-date-from-jan-2017-onwards.csv?raw=1'\n",
    "df = pd.read_csv(file_url)\n",
    "df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of `read_csv()` function is in Pandas' `DataFrame` type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`DataFrame` is Pandas' class that contains attributes and methods to work with a tabular data as shown above. Recall that we can create our own custom data type using the keyword `class` and define its attributes and methods. We can even override some of Python operators to work with our new data type. This is what Pandas library does with `DataFrame`. This `DataFrame` class provides some properties and methods that allows us to work with tabular data. \n",
    "\n",
    "For example, we can get all the name of the columns in our data frame using `df.columns` properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['month', 'town', 'flat_type', 'block', 'street_name', 'storey_range',\n",
       "       'floor_area_sqm', 'flat_model', 'lease_commence_date',\n",
       "       'remaining_lease', 'resale_price'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get the index (the names on the rows) using `df.index`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=95858, step=1)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also treat this data frame as a kind of matrix to find its shape using `df.shape`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(95858, 11)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the data contains 95858 rows and 11 columns. One of the column names is called `resale_price`. Since our aim is to predict the house price, this column is usually called the **target**. The rest of the columns is called the **features**. This means that we have about 10 feature columns.\n",
    "\n",
    "The idea supervised machine learning is that using the given data such as shown above, the computer would like to predict what is the *target* give a new set of *features*. The computer does this by *learning* the existing labelled data. The label in this case is the resale price from the historical sales data. \n",
    "\n",
    "In order to understand the data, it is important to be able to manipulate and work on the data frame. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Frame Operations\n",
    "\n",
    "It is important to know how to manipulate the data. Pandas has two data structures:\n",
    "- [Series](https://pandas.pydata.org/docs/user_guide/dsintro.html#series)\n",
    "- [DataFrame](https://pandas.pydata.org/docs/user_guide/dsintro.html#dataframe)\n",
    "\n",
    "You can consider `Series` data as one-dimensional labelled array while `DataFrame` data as two-dimensional labelled data structure. For example, the table that we saw previously, which is the output of `read_csv()` function, is a `DataFrame` because it has both rows and columns and, therefore, two dimensional. On the other hand, we can access just one of the columns from the data frame to get a `Series` data. \n",
    "\n",
    "### Getting a Column or a Row as a Series\n",
    "\n",
    "You can access the column data as series using the square bracket operator.\n",
    "\n",
    "```python\n",
    "df[column_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0        232000.0\n",
      "1        250000.0\n",
      "2        262000.0\n",
      "3        265000.0\n",
      "4        265000.0\n",
      "           ...   \n",
      "95853    650000.0\n",
      "95854    645000.0\n",
      "95855    585000.0\n",
      "95856    675000.0\n",
      "95857    625000.0\n",
      "Name: resale_price, Length: 95858, dtype: float64\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "print(df['resale_price'])\n",
    "print(type(df['resale_price']))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code above prints the column `resale_price` and its type. As can be seen the type of the output is a `Series` data type. \n",
    "\n",
    "You can also get some particular row by specifying its `index`. \n",
    "\n",
    "You can also access the column using the `.loc[index, column]` method. In this method, you need to specify the labels of the index. For example, to access all the rows for a particular column called `resale_price`, we can do as follows. Notice that we use `:` to access all the rows. Moreover, we specify the name of the columns in the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0        232000.0\n",
      "1        250000.0\n",
      "2        262000.0\n",
      "3        265000.0\n",
      "4        265000.0\n",
      "           ...   \n",
      "95853    650000.0\n",
      "95854    645000.0\n",
      "95855    585000.0\n",
      "95856    675000.0\n",
      "95857    625000.0\n",
      "Name: resale_price, Length: 95858, dtype: float64\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "print(df.loc[:, 'resale_price'])\n",
    "print(type(df.loc[:, 'resale_price']))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above code, we set the index to access all rows by using `:`. Recall that in Python's list slicing, we also use `:` to access all the element. Similarly here, we use `:` to access all the rows. In a similar way, we can use `:` to access all the columns, e.g. `df.loc[:, :]` will copy the whole data frame. \n",
    "\n",
    "This also gives you a hint how to access a particular row. Let's say, you only want to acces the first row, you can type the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "month                             2017-01\n",
      "town                           ANG MO KIO\n",
      "flat_type                          2 ROOM\n",
      "block                                 406\n",
      "street_name             ANG MO KIO AVE 10\n",
      "storey_range                     10 TO 12\n",
      "floor_area_sqm                         44\n",
      "flat_model                       Improved\n",
      "lease_commence_date                  1979\n",
      "remaining_lease        61 years 04 months\n",
      "resale_price                       232000\n",
      "Name: 0, dtype: object\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "print(df.loc[0, :])\n",
    "print(type(df.loc[0, :]))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above code, we access the first row, which is at index 0, and all the columns. \n",
    "\n",
    "Recall that all these data are of the type `Series`. You can create a Data Frame from an existing series just like when you create any other object by instantiating a `DataFrame` object and passing on an argument as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>month</th>\n",
       "      <td>2017-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>town</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flat_type</th>\n",
       "      <td>2 ROOM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>block</th>\n",
       "      <td>406</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>street_name</th>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>storey_range</th>\n",
       "      <td>10 TO 12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>floor_area_sqm</th>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flat_model</th>\n",
       "      <td>Improved</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lease_commence_date</th>\n",
       "      <td>1979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>remaining_lease</th>\n",
       "      <td>61 years 04 months</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>resale_price</th>\n",
       "      <td>232000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      0\n",
       "month                           2017-01\n",
       "town                         ANG MO KIO\n",
       "flat_type                        2 ROOM\n",
       "block                               406\n",
       "street_name           ANG MO KIO AVE 10\n",
       "storey_range                   10 TO 12\n",
       "floor_area_sqm                       44\n",
       "flat_model                     Improved\n",
       "lease_commence_date                1979\n",
       "remaining_lease      61 years 04 months\n",
       "resale_price                     232000"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_row0 = pd.DataFrame(df.loc[0, :])\n",
    "df_row0"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Rows and Columns as DataFrame\n",
    "\n",
    "The operator `:` works similar to Python's slicing. This means that you can get some rows by slicing them. For example, you can access the first 10 rows as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      month        town flat_type block        street_name storey_range  \\\n",
      "0   2017-01  ANG MO KIO    2 ROOM   406  ANG MO KIO AVE 10     10 TO 12   \n",
      "1   2017-01  ANG MO KIO    3 ROOM   108   ANG MO KIO AVE 4     01 TO 03   \n",
      "2   2017-01  ANG MO KIO    3 ROOM   602   ANG MO KIO AVE 5     01 TO 03   \n",
      "3   2017-01  ANG MO KIO    3 ROOM   465  ANG MO KIO AVE 10     04 TO 06   \n",
      "4   2017-01  ANG MO KIO    3 ROOM   601   ANG MO KIO AVE 5     01 TO 03   \n",
      "5   2017-01  ANG MO KIO    3 ROOM   150   ANG MO KIO AVE 5     01 TO 03   \n",
      "6   2017-01  ANG MO KIO    3 ROOM   447  ANG MO KIO AVE 10     04 TO 06   \n",
      "7   2017-01  ANG MO KIO    3 ROOM   218   ANG MO KIO AVE 1     04 TO 06   \n",
      "8   2017-01  ANG MO KIO    3 ROOM   447  ANG MO KIO AVE 10     04 TO 06   \n",
      "9   2017-01  ANG MO KIO    3 ROOM   571   ANG MO KIO AVE 3     01 TO 03   \n",
      "10  2017-01  ANG MO KIO    3 ROOM   534  ANG MO KIO AVE 10     01 TO 03   \n",
      "\n",
      "    floor_area_sqm      flat_model  lease_commence_date     remaining_lease  \\\n",
      "0             44.0        Improved                 1979  61 years 04 months   \n",
      "1             67.0  New Generation                 1978  60 years 07 months   \n",
      "2             67.0  New Generation                 1980  62 years 05 months   \n",
      "3             68.0  New Generation                 1980   62 years 01 month   \n",
      "4             67.0  New Generation                 1980  62 years 05 months   \n",
      "5             68.0  New Generation                 1981            63 years   \n",
      "6             68.0  New Generation                 1979  61 years 06 months   \n",
      "7             67.0  New Generation                 1976  58 years 04 months   \n",
      "8             68.0  New Generation                 1979  61 years 06 months   \n",
      "9             67.0  New Generation                 1979  61 years 04 months   \n",
      "10            68.0  New Generation                 1980   62 years 01 month   \n",
      "\n",
      "    resale_price  \n",
      "0       232000.0  \n",
      "1       250000.0  \n",
      "2       262000.0  \n",
      "3       265000.0  \n",
      "4       265000.0  \n",
      "5       275000.0  \n",
      "6       280000.0  \n",
      "7       285000.0  \n",
      "8       285000.0  \n",
      "9       285000.0  \n",
      "10      288500.0  \n",
      "<class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "print(df.loc[0:10, :])\n",
    "print(type(df.loc[0:10, :]))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice, however, that the slicing in Pandas' data frame is **inclusive** of the ending index unlike Python's slicing. The other thing to note about is that the output data type is no longer a series but rather a `DataFrame`. The reason is that now the data is two-dimensionsional. \n",
    "\n",
    "You can specify both the rows and the columns you want as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>month</th>\n",
       "      <th>town</th>\n",
       "      <th>flat_type</th>\n",
       "      <th>block</th>\n",
       "      <th>street_name</th>\n",
       "      <th>storey_range</th>\n",
       "      <th>floor_area_sqm</th>\n",
       "      <th>flat_model</th>\n",
       "      <th>lease_commence_date</th>\n",
       "      <th>remaining_lease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>2 ROOM</td>\n",
       "      <td>406</td>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "      <td>10 TO 12</td>\n",
       "      <td>44.0</td>\n",
       "      <td>Improved</td>\n",
       "      <td>1979</td>\n",
       "      <td>61 years 04 months</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>108</td>\n",
       "      <td>ANG MO KIO AVE 4</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1978</td>\n",
       "      <td>60 years 07 months</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>602</td>\n",
       "      <td>ANG MO KIO AVE 5</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1980</td>\n",
       "      <td>62 years 05 months</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>465</td>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "      <td>04 TO 06</td>\n",
       "      <td>68.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1980</td>\n",
       "      <td>62 years 01 month</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>601</td>\n",
       "      <td>ANG MO KIO AVE 5</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1980</td>\n",
       "      <td>62 years 05 months</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>150</td>\n",
       "      <td>ANG MO KIO AVE 5</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>68.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1981</td>\n",
       "      <td>63 years</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>447</td>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "      <td>04 TO 06</td>\n",
       "      <td>68.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1979</td>\n",
       "      <td>61 years 06 months</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>218</td>\n",
       "      <td>ANG MO KIO AVE 1</td>\n",
       "      <td>04 TO 06</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1976</td>\n",
       "      <td>58 years 04 months</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>447</td>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "      <td>04 TO 06</td>\n",
       "      <td>68.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1979</td>\n",
       "      <td>61 years 06 months</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>571</td>\n",
       "      <td>ANG MO KIO AVE 3</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1979</td>\n",
       "      <td>61 years 04 months</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>534</td>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>68.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1980</td>\n",
       "      <td>62 years 01 month</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      month        town flat_type block        street_name storey_range  \\\n",
       "0   2017-01  ANG MO KIO    2 ROOM   406  ANG MO KIO AVE 10     10 TO 12   \n",
       "1   2017-01  ANG MO KIO    3 ROOM   108   ANG MO KIO AVE 4     01 TO 03   \n",
       "2   2017-01  ANG MO KIO    3 ROOM   602   ANG MO KIO AVE 5     01 TO 03   \n",
       "3   2017-01  ANG MO KIO    3 ROOM   465  ANG MO KIO AVE 10     04 TO 06   \n",
       "4   2017-01  ANG MO KIO    3 ROOM   601   ANG MO KIO AVE 5     01 TO 03   \n",
       "5   2017-01  ANG MO KIO    3 ROOM   150   ANG MO KIO AVE 5     01 TO 03   \n",
       "6   2017-01  ANG MO KIO    3 ROOM   447  ANG MO KIO AVE 10     04 TO 06   \n",
       "7   2017-01  ANG MO KIO    3 ROOM   218   ANG MO KIO AVE 1     04 TO 06   \n",
       "8   2017-01  ANG MO KIO    3 ROOM   447  ANG MO KIO AVE 10     04 TO 06   \n",
       "9   2017-01  ANG MO KIO    3 ROOM   571   ANG MO KIO AVE 3     01 TO 03   \n",
       "10  2017-01  ANG MO KIO    3 ROOM   534  ANG MO KIO AVE 10     01 TO 03   \n",
       "\n",
       "    floor_area_sqm      flat_model  lease_commence_date     remaining_lease  \n",
       "0             44.0        Improved                 1979  61 years 04 months  \n",
       "1             67.0  New Generation                 1978  60 years 07 months  \n",
       "2             67.0  New Generation                 1980  62 years 05 months  \n",
       "3             68.0  New Generation                 1980   62 years 01 month  \n",
       "4             67.0  New Generation                 1980  62 years 05 months  \n",
       "5             68.0  New Generation                 1981            63 years  \n",
       "6             68.0  New Generation                 1979  61 years 06 months  \n",
       "7             67.0  New Generation                 1976  58 years 04 months  \n",
       "8             68.0  New Generation                 1979  61 years 06 months  \n",
       "9             67.0  New Generation                 1979  61 years 04 months  \n",
       "10            68.0  New Generation                 1980   62 years 01 month  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0:10,'month':'remaining_lease']"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to select the column, you can pass on a list of columns as shown in the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>town</th>\n",
       "      <th>block</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>406</td>\n",
       "      <td>232000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>108</td>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>602</td>\n",
       "      <td>262000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>465</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>601</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95853</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>326</td>\n",
       "      <td>650000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95854</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>360</td>\n",
       "      <td>645000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95855</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>326</td>\n",
       "      <td>585000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95856</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>355</td>\n",
       "      <td>675000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95857</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>277</td>\n",
       "      <td>625000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>95858 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             town block  resale_price\n",
       "0      ANG MO KIO   406      232000.0\n",
       "1      ANG MO KIO   108      250000.0\n",
       "2      ANG MO KIO   602      262000.0\n",
       "3      ANG MO KIO   465      265000.0\n",
       "4      ANG MO KIO   601      265000.0\n",
       "...           ...   ...           ...\n",
       "95853      YISHUN   326      650000.0\n",
       "95854      YISHUN   360      645000.0\n",
       "95855      YISHUN   326      585000.0\n",
       "95856      YISHUN   355      675000.0\n",
       "95857      YISHUN   277      625000.0\n",
       "\n",
       "[95858 rows x 3 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = ['town', 'block', 'resale_price']\n",
    "df.loc[:, columns]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A similar output can be obtained without `.loc`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>town</th>\n",
       "      <th>block</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>406</td>\n",
       "      <td>232000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>108</td>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>602</td>\n",
       "      <td>262000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>465</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>601</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95853</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>326</td>\n",
       "      <td>650000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95854</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>360</td>\n",
       "      <td>645000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95855</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>326</td>\n",
       "      <td>585000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95856</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>355</td>\n",
       "      <td>675000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95857</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>277</td>\n",
       "      <td>625000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>95858 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             town block  resale_price\n",
       "0      ANG MO KIO   406      232000.0\n",
       "1      ANG MO KIO   108      250000.0\n",
       "2      ANG MO KIO   602      262000.0\n",
       "3      ANG MO KIO   465      265000.0\n",
       "4      ANG MO KIO   601      265000.0\n",
       "...           ...   ...           ...\n",
       "95853      YISHUN   326      650000.0\n",
       "95854      YISHUN   360      645000.0\n",
       "95855      YISHUN   326      585000.0\n",
       "95856      YISHUN   355      675000.0\n",
       "95857      YISHUN   277      625000.0\n",
       "\n",
       "[95858 rows x 3 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[columns]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also combine specifying the rows and the columns as usual using `.loc`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>town</th>\n",
       "      <th>block</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>406</td>\n",
       "      <td>232000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>108</td>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>602</td>\n",
       "      <td>262000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>465</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>601</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>150</td>\n",
       "      <td>275000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>447</td>\n",
       "      <td>280000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>218</td>\n",
       "      <td>285000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>447</td>\n",
       "      <td>285000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>571</td>\n",
       "      <td>285000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>534</td>\n",
       "      <td>288500.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          town block  resale_price\n",
       "0   ANG MO KIO   406      232000.0\n",
       "1   ANG MO KIO   108      250000.0\n",
       "2   ANG MO KIO   602      262000.0\n",
       "3   ANG MO KIO   465      265000.0\n",
       "4   ANG MO KIO   601      265000.0\n",
       "5   ANG MO KIO   150      275000.0\n",
       "6   ANG MO KIO   447      280000.0\n",
       "7   ANG MO KIO   218      285000.0\n",
       "8   ANG MO KIO   447      285000.0\n",
       "9   ANG MO KIO   571      285000.0\n",
       "10  ANG MO KIO   534      288500.0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0:10, columns]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The index is not always necessarily be an integer. Pandas can take strings as the index of a data frame. But there are times, even when the index is not an integer, we still prefer to locate using the position of the rows to select. In this case, we can use `.iloc[position_index, position_column]`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>town</th>\n",
       "      <th>block</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>406</td>\n",
       "      <td>232000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>108</td>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>602</td>\n",
       "      <td>262000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>465</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>601</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>150</td>\n",
       "      <td>275000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>447</td>\n",
       "      <td>280000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>218</td>\n",
       "      <td>285000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>447</td>\n",
       "      <td>285000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>571</td>\n",
       "      <td>285000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         town block  resale_price\n",
       "0  ANG MO KIO   406      232000.0\n",
       "1  ANG MO KIO   108      250000.0\n",
       "2  ANG MO KIO   602      262000.0\n",
       "3  ANG MO KIO   465      265000.0\n",
       "4  ANG MO KIO   601      265000.0\n",
       "5  ANG MO KIO   150      275000.0\n",
       "6  ANG MO KIO   447      280000.0\n",
       "7  ANG MO KIO   218      285000.0\n",
       "8  ANG MO KIO   447      285000.0\n",
       "9  ANG MO KIO   571      285000.0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = [1, 3, -1]\n",
    "df.iloc[0:10, columns]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code gives the same data frame but it uses different input to specifies. By using `.iloc[]`, we specify the position of the index and the columns instead of the label of the index and the columns. It happens that for the index, the position numbering is exactly the same as the label. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting Data Using Conditions\n",
    "\n",
    "We can use conditions with Pandas' data frame to select particular rows and columns using either `.loc[]` or `.iloc[]`. The reason is that these methods can take in boolean arrays.\n",
    "\n",
    "Let's see some examples below. First, let's list down the resale price by focusing on the block at a given town. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>town</th>\n",
       "      <th>block</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>406</td>\n",
       "      <td>232000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>108</td>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>602</td>\n",
       "      <td>262000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>465</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>601</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95853</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>326</td>\n",
       "      <td>650000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95854</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>360</td>\n",
       "      <td>645000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95855</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>326</td>\n",
       "      <td>585000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95856</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>355</td>\n",
       "      <td>675000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95857</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>277</td>\n",
       "      <td>625000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>95858 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             town block  resale_price\n",
       "0      ANG MO KIO   406      232000.0\n",
       "1      ANG MO KIO   108      250000.0\n",
       "2      ANG MO KIO   602      262000.0\n",
       "3      ANG MO KIO   465      265000.0\n",
       "4      ANG MO KIO   601      265000.0\n",
       "...           ...   ...           ...\n",
       "95853      YISHUN   326      650000.0\n",
       "95854      YISHUN   360      645000.0\n",
       "95855      YISHUN   326      585000.0\n",
       "95856      YISHUN   355      675000.0\n",
       "95857      YISHUN   277      625000.0\n",
       "\n",
       "[95858 rows x 3 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = ['town', 'block', 'resale_price']\n",
    "df.loc[:, columns]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we want to see those sales where the price is greater than \\$500k. We can put in this condition in filtering the rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>town</th>\n",
       "      <th>block</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>304</td>\n",
       "      <td>518000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>646</td>\n",
       "      <td>518000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>328</td>\n",
       "      <td>560000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>588C</td>\n",
       "      <td>688000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>588D</td>\n",
       "      <td>730000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95853</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>326</td>\n",
       "      <td>650000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95854</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>360</td>\n",
       "      <td>645000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95855</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>326</td>\n",
       "      <td>585000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95856</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>355</td>\n",
       "      <td>675000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95857</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>277</td>\n",
       "      <td>625000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>27233 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             town block  resale_price\n",
       "43     ANG MO KIO   304      518000.0\n",
       "44     ANG MO KIO   646      518000.0\n",
       "45     ANG MO KIO   328      560000.0\n",
       "46     ANG MO KIO  588C      688000.0\n",
       "47     ANG MO KIO  588D      730000.0\n",
       "...           ...   ...           ...\n",
       "95853      YISHUN   326      650000.0\n",
       "95854      YISHUN   360      645000.0\n",
       "95855      YISHUN   326      585000.0\n",
       "95856      YISHUN   355      675000.0\n",
       "95857      YISHUN   277      625000.0\n",
       "\n",
       "[27233 rows x 3 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[df['resale_price'] > 500_000, columns]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Python ignores the underscores in between numeric literals and you can use it to make it easier to read. \n",
    "\n",
    "Let's say if we want to find all those sales between \\\\$500k and \\\\$600k only, we can use the AND operator `&` to have more than one conditions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>town</th>\n",
       "      <th>block</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>304</td>\n",
       "      <td>518000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>646</td>\n",
       "      <td>518000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>328</td>\n",
       "      <td>560000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>101</td>\n",
       "      <td>500000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>BEDOK</td>\n",
       "      <td>185</td>\n",
       "      <td>580000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95849</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>504C</td>\n",
       "      <td>550000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95850</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>511B</td>\n",
       "      <td>600000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95851</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>504C</td>\n",
       "      <td>590000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95852</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>838</td>\n",
       "      <td>571888.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95855</th>\n",
       "      <td>YISHUN</td>\n",
       "      <td>326</td>\n",
       "      <td>585000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13478 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             town block  resale_price\n",
       "43     ANG MO KIO   304      518000.0\n",
       "44     ANG MO KIO   646      518000.0\n",
       "45     ANG MO KIO   328      560000.0\n",
       "49     ANG MO KIO   101      500000.0\n",
       "110         BEDOK   185      580000.0\n",
       "...           ...   ...           ...\n",
       "95849      YISHUN  504C      550000.0\n",
       "95850      YISHUN  511B      600000.0\n",
       "95851      YISHUN  504C      590000.0\n",
       "95852      YISHUN   838      571888.0\n",
       "95855      YISHUN   326      585000.0\n",
       "\n",
       "[13478 rows x 3 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[(df['resale_price'] >= 500_000) & (df['resale_price'] <= 600_000), columns]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note: the parenthesis separating the two AND conditions are compulsory.**\n",
    "\n",
    "We can also specify more conditions. For example, we are only interested in ANG MO KIO area. We can have the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>town</th>\n",
       "      <th>block</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>304</td>\n",
       "      <td>518000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>646</td>\n",
       "      <td>518000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>328</td>\n",
       "      <td>560000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>101</td>\n",
       "      <td>500000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1219</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>351</td>\n",
       "      <td>530000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94741</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>545</td>\n",
       "      <td>590000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94742</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>545</td>\n",
       "      <td>600000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94743</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>551</td>\n",
       "      <td>520000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94746</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>642</td>\n",
       "      <td>545000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94749</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>353</td>\n",
       "      <td>588000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>329 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             town block  resale_price\n",
       "43     ANG MO KIO   304      518000.0\n",
       "44     ANG MO KIO   646      518000.0\n",
       "45     ANG MO KIO   328      560000.0\n",
       "49     ANG MO KIO   101      500000.0\n",
       "1219   ANG MO KIO   351      530000.0\n",
       "...           ...   ...           ...\n",
       "94741  ANG MO KIO   545      590000.0\n",
       "94742  ANG MO KIO   545      600000.0\n",
       "94743  ANG MO KIO   551      520000.0\n",
       "94746  ANG MO KIO   642      545000.0\n",
       "94749  ANG MO KIO   353      588000.0\n",
       "\n",
       "[329 rows x 3 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[(df['resale_price'] >= 500_000) & (df['resale_price'] <= 600_000) &\n",
    "       (df['town'] == 'ANG MO KIO'), columns]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are interested only in blocks 300s and 400s, you can add this conditions further."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>town</th>\n",
       "      <th>block</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>304</td>\n",
       "      <td>518000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>328</td>\n",
       "      <td>560000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1219</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>351</td>\n",
       "      <td>530000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2337</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>344</td>\n",
       "      <td>538000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2338</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>329</td>\n",
       "      <td>548000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92327</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>353</td>\n",
       "      <td>520000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92332</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>459</td>\n",
       "      <td>600000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92335</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>459</td>\n",
       "      <td>500000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94740</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>305</td>\n",
       "      <td>537000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94749</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>353</td>\n",
       "      <td>588000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>174 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             town block  resale_price\n",
       "43     ANG MO KIO   304      518000.0\n",
       "45     ANG MO KIO   328      560000.0\n",
       "1219   ANG MO KIO   351      530000.0\n",
       "2337   ANG MO KIO   344      538000.0\n",
       "2338   ANG MO KIO   329      548000.0\n",
       "...           ...   ...           ...\n",
       "92327  ANG MO KIO   353      520000.0\n",
       "92332  ANG MO KIO   459      600000.0\n",
       "92335  ANG MO KIO   459      500000.0\n",
       "94740  ANG MO KIO   305      537000.0\n",
       "94749  ANG MO KIO   353      588000.0\n",
       "\n",
       "[174 rows x 3 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[(df['resale_price'] >= 500_000) & (df['resale_price'] <= 600_000) &\n",
    "       (df['town'] == 'ANG MO KIO') & \n",
    "       (df['block'] >= '300') & (df['block'] < '500'), columns]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Series and DataFrame Functions\n",
    "\n",
    "Pandas also provides several functions that can be useful in understanding the data. In this section, we will explore some of these.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating DataFrame and Series\n",
    "\n",
    "We can create a new DataFrame from other data type such as dictionary, list-like objects, or Series. For example, given a `Series`, you can convert into a `DataFrame` as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>232000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>262000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95853</th>\n",
       "      <td>650000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95854</th>\n",
       "      <td>645000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95855</th>\n",
       "      <td>585000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95856</th>\n",
       "      <td>675000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95857</th>\n",
       "      <td>625000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>95858 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       resale_price\n",
       "0          232000.0\n",
       "1          250000.0\n",
       "2          262000.0\n",
       "3          265000.0\n",
       "4          265000.0\n",
       "...             ...\n",
       "95853      650000.0\n",
       "95854      645000.0\n",
       "95855      585000.0\n",
       "95856      675000.0\n",
       "95857      625000.0\n",
       "\n",
       "[95858 rows x 1 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "price = df['resale_price']\n",
    "print(isinstance(price, pd.Series))\n",
    "price_df = pd.DataFrame(price)\n",
    "print(isinstance(price_df, pd.DataFrame))\n",
    "price_df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, you can convert other data to a `Series` by using its contructor. In the example below, we create a new series from a list of integers from 2 to 100. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0       2\n",
       "1       3\n",
       "2       4\n",
       "3       5\n",
       "4       6\n",
       "     ... \n",
       "94     96\n",
       "95     97\n",
       "96     98\n",
       "97     99\n",
       "98    100\n",
       "Length: 99, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_series = pd.Series(list(range(2,101)))\n",
    "print(isinstance(new_series, pd.Series))\n",
    "new_series"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copying\n",
    "\n",
    "One useful function is to copy a data frame to another dataframe. We can use `df.copy()`. This function has an argument `deep` which by default is `True`. If it is true, it will do a deep copy of the Data Frame. Otherwise, it will just do a shallow copy. See [documention](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.copy.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>month</th>\n",
       "      <th>town</th>\n",
       "      <th>flat_type</th>\n",
       "      <th>block</th>\n",
       "      <th>street_name</th>\n",
       "      <th>storey_range</th>\n",
       "      <th>floor_area_sqm</th>\n",
       "      <th>flat_model</th>\n",
       "      <th>lease_commence_date</th>\n",
       "      <th>remaining_lease</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>2 ROOM</td>\n",
       "      <td>406</td>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "      <td>10 TO 12</td>\n",
       "      <td>44.0</td>\n",
       "      <td>Improved</td>\n",
       "      <td>1979</td>\n",
       "      <td>61 years 04 months</td>\n",
       "      <td>232000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>108</td>\n",
       "      <td>ANG MO KIO AVE 4</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1978</td>\n",
       "      <td>60 years 07 months</td>\n",
       "      <td>250000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>602</td>\n",
       "      <td>ANG MO KIO AVE 5</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1980</td>\n",
       "      <td>62 years 05 months</td>\n",
       "      <td>262000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>465</td>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "      <td>04 TO 06</td>\n",
       "      <td>68.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1980</td>\n",
       "      <td>62 years 01 month</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>3 ROOM</td>\n",
       "      <td>601</td>\n",
       "      <td>ANG MO KIO AVE 5</td>\n",
       "      <td>01 TO 03</td>\n",
       "      <td>67.0</td>\n",
       "      <td>New Generation</td>\n",
       "      <td>1980</td>\n",
       "      <td>62 years 05 months</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95853</th>\n",
       "      <td>2021-04</td>\n",
       "      <td>YISHUN</td>\n",
       "      <td>EXECUTIVE</td>\n",
       "      <td>326</td>\n",
       "      <td>YISHUN RING RD</td>\n",
       "      <td>10 TO 12</td>\n",
       "      <td>146.0</td>\n",
       "      <td>Maisonette</td>\n",
       "      <td>1988</td>\n",
       "      <td>66 years 04 months</td>\n",
       "      <td>650000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95854</th>\n",
       "      <td>2021-04</td>\n",
       "      <td>YISHUN</td>\n",
       "      <td>EXECUTIVE</td>\n",
       "      <td>360</td>\n",
       "      <td>YISHUN RING RD</td>\n",
       "      <td>04 TO 06</td>\n",
       "      <td>146.0</td>\n",
       "      <td>Maisonette</td>\n",
       "      <td>1988</td>\n",
       "      <td>66 years 04 months</td>\n",
       "      <td>645000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95855</th>\n",
       "      <td>2021-04</td>\n",
       "      <td>YISHUN</td>\n",
       "      <td>EXECUTIVE</td>\n",
       "      <td>326</td>\n",
       "      <td>YISHUN RING RD</td>\n",
       "      <td>10 TO 12</td>\n",
       "      <td>146.0</td>\n",
       "      <td>Maisonette</td>\n",
       "      <td>1988</td>\n",
       "      <td>66 years 04 months</td>\n",
       "      <td>585000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95856</th>\n",
       "      <td>2021-04</td>\n",
       "      <td>YISHUN</td>\n",
       "      <td>EXECUTIVE</td>\n",
       "      <td>355</td>\n",
       "      <td>YISHUN RING RD</td>\n",
       "      <td>10 TO 12</td>\n",
       "      <td>146.0</td>\n",
       "      <td>Maisonette</td>\n",
       "      <td>1988</td>\n",
       "      <td>66 years 08 months</td>\n",
       "      <td>675000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95857</th>\n",
       "      <td>2021-04</td>\n",
       "      <td>YISHUN</td>\n",
       "      <td>EXECUTIVE</td>\n",
       "      <td>277</td>\n",
       "      <td>YISHUN ST 22</td>\n",
       "      <td>04 TO 06</td>\n",
       "      <td>146.0</td>\n",
       "      <td>Maisonette</td>\n",
       "      <td>1985</td>\n",
       "      <td>63 years 05 months</td>\n",
       "      <td>625000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>95858 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         month        town  flat_type block        street_name storey_range  \\\n",
       "0      2017-01  ANG MO KIO     2 ROOM   406  ANG MO KIO AVE 10     10 TO 12   \n",
       "1      2017-01  ANG MO KIO     3 ROOM   108   ANG MO KIO AVE 4     01 TO 03   \n",
       "2      2017-01  ANG MO KIO     3 ROOM   602   ANG MO KIO AVE 5     01 TO 03   \n",
       "3      2017-01  ANG MO KIO     3 ROOM   465  ANG MO KIO AVE 10     04 TO 06   \n",
       "4      2017-01  ANG MO KIO     3 ROOM   601   ANG MO KIO AVE 5     01 TO 03   \n",
       "...        ...         ...        ...   ...                ...          ...   \n",
       "95853  2021-04      YISHUN  EXECUTIVE   326     YISHUN RING RD     10 TO 12   \n",
       "95854  2021-04      YISHUN  EXECUTIVE   360     YISHUN RING RD     04 TO 06   \n",
       "95855  2021-04      YISHUN  EXECUTIVE   326     YISHUN RING RD     10 TO 12   \n",
       "95856  2021-04      YISHUN  EXECUTIVE   355     YISHUN RING RD     10 TO 12   \n",
       "95857  2021-04      YISHUN  EXECUTIVE   277       YISHUN ST 22     04 TO 06   \n",
       "\n",
       "       floor_area_sqm      flat_model  lease_commence_date  \\\n",
       "0                44.0        Improved                 1979   \n",
       "1                67.0  New Generation                 1978   \n",
       "2                67.0  New Generation                 1980   \n",
       "3                68.0  New Generation                 1980   \n",
       "4                67.0  New Generation                 1980   \n",
       "...               ...             ...                  ...   \n",
       "95853           146.0      Maisonette                 1988   \n",
       "95854           146.0      Maisonette                 1988   \n",
       "95855           146.0      Maisonette                 1988   \n",
       "95856           146.0      Maisonette                 1988   \n",
       "95857           146.0      Maisonette                 1985   \n",
       "\n",
       "          remaining_lease  resale_price  \n",
       "0      61 years 04 months      232000.0  \n",
       "1      60 years 07 months      250000.0  \n",
       "2      62 years 05 months      262000.0  \n",
       "3       62 years 01 month      265000.0  \n",
       "4      62 years 05 months      265000.0  \n",
       "...                   ...           ...  \n",
       "95853  66 years 04 months      650000.0  \n",
       "95854  66 years 04 months      645000.0  \n",
       "95855  66 years 04 months      585000.0  \n",
       "95856  66 years 08 months      675000.0  \n",
       "95857  63 years 05 months      625000.0  \n",
       "\n",
       "[95858 rows x 11 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = df.copy()\n",
    "df2"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistical Functions\n",
    "\n",
    "We can get some descriptive statistics about the data using some of Pandas functions. For example, we can get the five point summary using `.describe()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>floor_area_sqm</th>\n",
       "      <th>lease_commence_date</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>95858.000000</td>\n",
       "      <td>95858.000000</td>\n",
       "      <td>9.585800e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>97.772234</td>\n",
       "      <td>1994.553934</td>\n",
       "      <td>4.467242e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>24.238799</td>\n",
       "      <td>13.128913</td>\n",
       "      <td>1.552974e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>31.000000</td>\n",
       "      <td>1966.000000</td>\n",
       "      <td>1.400000e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>82.000000</td>\n",
       "      <td>1984.000000</td>\n",
       "      <td>3.350000e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>95.000000</td>\n",
       "      <td>1995.000000</td>\n",
       "      <td>4.160000e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>113.000000</td>\n",
       "      <td>2004.000000</td>\n",
       "      <td>5.250000e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>249.000000</td>\n",
       "      <td>2019.000000</td>\n",
       "      <td>1.258000e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       floor_area_sqm  lease_commence_date  resale_price\n",
       "count    95858.000000         95858.000000  9.585800e+04\n",
       "mean        97.772234          1994.553934  4.467242e+05\n",
       "std         24.238799            13.128913  1.552974e+05\n",
       "min         31.000000          1966.000000  1.400000e+05\n",
       "25%         82.000000          1984.000000  3.350000e+05\n",
       "50%         95.000000          1995.000000  4.160000e+05\n",
       "75%        113.000000          2004.000000  5.250000e+05\n",
       "max        249.000000          2019.000000  1.258000e+06"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code only shows a few columns because the other columns are not numbers. Pandas will only try to get the statistics of the columns that contain numeric numbers. We can also get the individual statistical functions as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "446724.22886801313\n"
     ]
    }
   ],
   "source": [
    "print(df['resale_price'].mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "155297.43748684428\n"
     ]
    }
   ],
   "source": [
    "print(df['resale_price'].std())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "140000.0\n"
     ]
    }
   ],
   "source": [
    "print(df['resale_price'].min())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1258000.0\n"
     ]
    }
   ],
   "source": [
    "print(df['resale_price'].max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "525000.0\n"
     ]
    }
   ],
   "source": [
    "print(df['resale_price'].quantile(q=0.75))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can change the way the statistics is computed. Currently, the statistics is calculated over all the rows in the vertical dimension. This is what is considered as `axis=0` in Pandas. You can change it to compute over all the columns by specifying `axis=1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         78007.666667\n",
       "1         84015.000000\n",
       "2         88015.666667\n",
       "3         89016.000000\n",
       "4         89015.666667\n",
       "             ...      \n",
       "95853    217378.000000\n",
       "95854    215711.333333\n",
       "95855    195711.333333\n",
       "95856    225711.333333\n",
       "95857    209043.666667\n",
       "Length: 95858, dtype: float64"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.mean(axis=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, Pandas only computes the mean from the numeric data across the columns."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transposing Data Frame\n",
    "\n",
    "You can also change the rows into the column and the column into the rows. For example, previously we have this data frame we created from a `Series` when extracting row 0. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>month</th>\n",
       "      <td>2017-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>town</th>\n",
       "      <td>ANG MO KIO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flat_type</th>\n",
       "      <td>2 ROOM</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>block</th>\n",
       "      <td>406</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>street_name</th>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>storey_range</th>\n",
       "      <td>10 TO 12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>floor_area_sqm</th>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>flat_model</th>\n",
       "      <td>Improved</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>lease_commence_date</th>\n",
       "      <td>1979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>remaining_lease</th>\n",
       "      <td>61 years 04 months</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>resale_price</th>\n",
       "      <td>232000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      0\n",
       "month                           2017-01\n",
       "town                         ANG MO KIO\n",
       "flat_type                        2 ROOM\n",
       "block                               406\n",
       "street_name           ANG MO KIO AVE 10\n",
       "storey_range                   10 TO 12\n",
       "floor_area_sqm                       44\n",
       "flat_model                     Improved\n",
       "lease_commence_date                1979\n",
       "remaining_lease      61 years 04 months\n",
       "resale_price                     232000"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_row0"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above code, the column is row 0 and the rows are the different column names. You can transpose the data using the `.T` property."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>month</th>\n",
       "      <th>town</th>\n",
       "      <th>flat_type</th>\n",
       "      <th>block</th>\n",
       "      <th>street_name</th>\n",
       "      <th>storey_range</th>\n",
       "      <th>floor_area_sqm</th>\n",
       "      <th>flat_model</th>\n",
       "      <th>lease_commence_date</th>\n",
       "      <th>remaining_lease</th>\n",
       "      <th>resale_price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2017-01</td>\n",
       "      <td>ANG MO KIO</td>\n",
       "      <td>2 ROOM</td>\n",
       "      <td>406</td>\n",
       "      <td>ANG MO KIO AVE 10</td>\n",
       "      <td>10 TO 12</td>\n",
       "      <td>44</td>\n",
       "      <td>Improved</td>\n",
       "      <td>1979</td>\n",
       "      <td>61 years 04 months</td>\n",
       "      <td>232000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     month        town flat_type block        street_name storey_range  \\\n",
       "0  2017-01  ANG MO KIO    2 ROOM   406  ANG MO KIO AVE 10     10 TO 12   \n",
       "\n",
       "  floor_area_sqm flat_model lease_commence_date     remaining_lease  \\\n",
       "0             44   Improved                1979  61 years 04 months   \n",
       "\n",
       "  resale_price  \n",
       "0       232000  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_row0_transposed = df_row0.T\n",
    "df_row0_transposed"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vector Operations\n",
    "\n",
    "One useful function in Pandas is `.apply()` (see [documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html)) where we can apply some function to all the data in the column or row or Series in a vectorized manner. In this way, we need not iterate or loop the data one at a time to apply this computation. \n",
    "\n",
    "For example, if we want to create a column for resale price in terms of $1000, we can use the `.apply()` method by dividing the `resale_price` column with 1000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        232.0\n",
       "1        250.0\n",
       "2        262.0\n",
       "3        265.0\n",
       "4        265.0\n",
       "         ...  \n",
       "95853    650.0\n",
       "95854    645.0\n",
       "95855    585.0\n",
       "95856    675.0\n",
       "95857    625.0\n",
       "Name: resale_price_in1000, Length: 95858, dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def divide_by_1000(data):\n",
    "    return data / 1000\n",
    "\n",
    "df['resale_price_in1000'] = df['resale_price'].apply(divide_by_1000)\n",
    "df['resale_price_in1000']"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method `.apply()` takes in a function that will be processed for every data in that Series. Instead of creating a named function, we can make use of Python's lambda function to do the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        232.0\n",
       "1        250.0\n",
       "2        262.0\n",
       "3        265.0\n",
       "4        265.0\n",
       "         ...  \n",
       "95853    650.0\n",
       "95854    645.0\n",
       "95855    585.0\n",
       "95856    675.0\n",
       "95857    625.0\n",
       "Name: resale_price_in1000, Length: 95858, dtype: float64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['resale_price_in1000'] = df['resale_price'].apply(lambda data: data/1000)\n",
    "df['resale_price_in1000']"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the argument in `divide_by_1000()` becomes the first token after the keyword `lambda`. The return value of the function is provided after the colon, i.e. `:`. \n",
    "\n",
    "You can use this to process and create any other kind of data. For example, we can create a new categorical column called \"Pricey\" and set any sales above \\$500k is considered as pricey otherwise is not. If it is pricey, we will label it as 1, otherwise, as 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>resale_price_in1000</th>\n",
       "      <th>pricey</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>232.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>250.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>262.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>265.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>265.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95853</th>\n",
       "      <td>650.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95854</th>\n",
       "      <td>645.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95855</th>\n",
       "      <td>585.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95856</th>\n",
       "      <td>675.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95857</th>\n",
       "      <td>625.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>95858 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       resale_price_in1000  pricey\n",
       "0                    232.0       0\n",
       "1                    250.0       0\n",
       "2                    262.0       0\n",
       "3                    265.0       0\n",
       "4                    265.0       0\n",
       "...                    ...     ...\n",
       "95853                650.0       1\n",
       "95854                645.0       1\n",
       "95855                585.0       1\n",
       "95856                675.0       1\n",
       "95857                625.0       1\n",
       "\n",
       "[95858 rows x 2 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['pricey'] = df['resale_price_in1000'].apply(lambda price: 1 if price > 500 else 0 )\n",
    "df[['resale_price_in1000', 'pricey']]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above function, we use the if *expression* to specify the return value for the lambda function. It follows the following format: \n",
    "\n",
    "```python\n",
    "expression_if_true if condition else expression_if_false\n",
    "```\n",
    "\n",
    "There are many other Pandas functions and methods. It is recommended that you look into the documentation for further references."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "\n",
    "- [Pandas User Guide](https://pandas.pydata.org/docs/user_guide/index.html)\n",
    "- [Pandas API Reference](https://pandas.pydata.org/docs/reference/index.html) "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalization\n",
    "\n",
    "Many times, we will need to normalize the data, both the features and the target. The reason is that each column in the dataset may have different scales. For example, the column  `floor_area_sqm` is in between 33 to 249 while `lease_commense_date` is actually in a range between 1966 and 2019. See below statistics for these two columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    95858.000000\n",
       "mean        97.772234\n",
       "std         24.238799\n",
       "min         31.000000\n",
       "25%         82.000000\n",
       "50%         95.000000\n",
       "75%        113.000000\n",
       "max        249.000000\n",
       "Name: floor_area_sqm, dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "count    95858.000000\n",
       "mean      1994.553934\n",
       "std         13.128913\n",
       "min       1966.000000\n",
       "25%       1984.000000\n",
       "50%       1995.000000\n",
       "75%       2004.000000\n",
       "max       2019.000000\n",
       "Name: lease_commence_date, dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(df['floor_area_sqm'].describe())\n",
    "display(df['lease_commence_date'].describe())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we will see later in subsequent weeks, we usually need to normalize the data before doing any training for our machine learning model. There are two common normalization: \n",
    "- z normalization\n",
    "- minmax normalization\n",
    "\n",
    "You will work on the functions to calculate these normalization in your problem sets."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Z Normalization\n",
    "\n",
    "This is also called as standardization. In this tranformation, we move the mean of the data distribution to 0 and its standard deviation to 1. The equation is given as follows.\n",
    "\n",
    "$$normalized = \\frac{data - \\mu}{\\sigma}$$"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Min-Max Normalization\n",
    "\n",
    "In this transformation, we scale the data in such a way that the maximum value in the distribution is 1 and its minimum value is 0. We can scale it using the following equation.\n",
    "\n",
    "$$normalized = \\frac{data - min}{max - min}$$"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting Dataset"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One common pre-processing operations that we normally do in machine learning is to split the data into:\n",
    "- **training** dataset\n",
    "- **test** dataset\n",
    "\n",
    "The idea of splitting the dataset is simply because we should **NOT** use the same data to verify the model which we train. Let's illustrate this using our HDB resale price dataset. We have this HDB resale price dataset with 95858 entries. In our machine learning process, we would like to use this data to do the following:\n",
    "1. Train the model using the dataset\n",
    "1. Verify the accuracy of the model\n",
    "\n",
    "If we only have one dataset, we cannot use the same data to verify the accuracy with the ones we use to train the model. This bias would obviously create high accuracy. The analogy is like when a teacher giving exactly the same question during the exam as the ones during the practice session. \n",
    "\n",
    "To overcome this, we should split the data into two. One set is used to train the model while the other one is used to verify the model. Coming back to the analogy of the teacher giving the exam, if the teacher has a bank of questions, he or she should separate the questions into two. Some questions can be used for practice before the exam, while the rest can be used as exam questions. \n",
    "\n",
    "If we illustrate this using our HDB resale price dataset, this means that we have to split the table into two. Out of 95868 entries, we will take some entries as out training dataset and leave the rest for out test dataset. The common proportion is either 70% or 80% for the training dataset and leave the other 30% or 20% for the test dataset. \n",
    "\n",
    "One important note is that the split must be done **randomly**. This is to avoid systematic bias in the split of the dataset. For example, one may enter the data according to the flat type and so flat with smaller rooms and smaller floor area will be on the top rows and those with the larger flats will be somewhere at the end of the rows. If we do not split the data randomly, we may not have any larger flats in our training set and only use the smaller flats. Similarly it can happen with any other column such as the block or the town area. \n",
    "\n",
    "There are times in machine learning, we need to experiment with different parameters and find the optimum parameters. In these cases, the dataset is usually split into three:\n",
    "- **training** dataset, which is used to build the model\n",
    "- **validation** dataset, which is used to evaluate the model for various parameters and to choose the optimum parameter\n",
    "- **test** dataset, which is used to evaluate the model built with the optimum parameter found previously\n",
    "\n",
    "You will see some of these application in the subsequent weeks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.5 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "ac37b77c3c0f43e60bec193f0626743b91dd65d8d4aeca5713e457ab7e777e4c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}