Segmenting_and_Clustering_Neighborhoods_in_Toronto_2.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {
                "collapsed": true
            },
            "source": "# Segmenting and Clustering Neighborhoods in Toronto"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "**Harry Ngo**   "
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "**_Background_ _information_**  \nIn this assignment, you will be required to explore, segment, and cluster the neighborhoods in the city of Toronto. However, unlike New York, the neighborhood data is not readily available on the internet. What is interesting about the field of data science is that each project can be challenging in its unique way, so you need to learn to be agile and refine the skill to learn new libraries and tools quickly depending on the project.   \nFor the Toronto neighborhood data, a Wikipedia page exists that has all the information we need to explore and cluster the neighborhoods in Toronto. You will be required to scrape the Wikipedia page and wrangle the data, clean it, and then read it into a pandas dataframe so that it is in a structured format like the New York dataset.  \nOnce the data is in a structured format, you can replicate the analysis that we did to the New York City dataset to explore and cluster the neighborhoods in the city of Toronto."
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## Step 1: Read in and clean Wikipedia data"
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "metadata": {},
            "outputs": [],
            "source": "import pandas as pd"
        },
        {
            "cell_type": "code",
            "execution_count": 56,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Postal code</th>\n      <th>Borough</th>\n      <th>Neighborhood</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>M1A</td>\n      <td>Not assigned</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>M2A</td>\n      <td>Not assigned</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>M3A</td>\n      <td>North York</td>\n      <td>Parkwoods</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>M4A</td>\n      <td>North York</td>\n      <td>Victoria Village</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>M5A</td>\n      <td>Downtown Toronto</td>\n      <td>Regent Park / Harbourfront</td>\n    </tr>\n  </tbody>\n</table>\n</div>",
                        "text/plain": "  Postal code           Borough                Neighborhood\n0         M1A      Not assigned                         NaN\n1         M2A      Not assigned                         NaN\n2         M3A        North York                   Parkwoods\n3         M4A        North York            Victoria Village\n4         M5A  Downtown Toronto  Regent Park / Harbourfront"
                    },
                    "execution_count": 56,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "# Using pandas to read HTML\ndf = pd.read_html('https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M')[0]\ndf.head()"
        },
        {
            "cell_type": "code",
            "execution_count": 57,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>PostalCode</th>\n      <th>Borough</th>\n      <th>Neighborhood</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>2</th>\n      <td>M3A</td>\n      <td>North York</td>\n      <td>Parkwoods</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>M4A</td>\n      <td>North York</td>\n      <td>Victoria Village</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>M5A</td>\n      <td>Downtown Toronto</td>\n      <td>Regent Park / Harbourfront</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>M6A</td>\n      <td>North York</td>\n      <td>Lawrence Manor / Lawrence Heights</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>M7A</td>\n      <td>Downtown Toronto</td>\n      <td>Queen's Park / Ontario Provincial Government</td>\n    </tr>\n  </tbody>\n</table>\n</div>",
                        "text/plain": "  PostalCode           Borough                                  Neighborhood\n2        M3A        North York                                     Parkwoods\n3        M4A        North York                              Victoria Village\n4        M5A  Downtown Toronto                    Regent Park / Harbourfront\n5        M6A        North York             Lawrence Manor / Lawrence Heights\n6        M7A  Downtown Toronto  Queen's Park / Ontario Provincial Government"
                    },
                    "execution_count": 57,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "# Only process the cells that have an assigned borough. Ignore cells with a borough that is 'Not assigned'.\ndf = df[df['Borough'] != 'Not assigned']\n# Rename Postal code column\ndf = df.rename(columns={'Postal code': 'PostalCode'})\n\ndf.head()"
        },
        {
            "cell_type": "code",
            "execution_count": 58,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>PostalCode</th>\n      <th>Borough</th>\n      <th>Neighborhood</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>2</th>\n      <td>M3A</td>\n      <td>North York</td>\n      <td>Parkwoods</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>M4A</td>\n      <td>North York</td>\n      <td>Victoria Village</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>M5A</td>\n      <td>Downtown Toronto</td>\n      <td>Regent Park, Harbourfront</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>M6A</td>\n      <td>North York</td>\n      <td>Lawrence Manor, Lawrence Heights</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>M7A</td>\n      <td>Downtown Toronto</td>\n      <td>Queen's Park, Ontario Provincial Government</td>\n    </tr>\n  </tbody>\n</table>\n</div>",
                        "text/plain": "  PostalCode           Borough                                 Neighborhood\n2        M3A        North York                                    Parkwoods\n3        M4A        North York                             Victoria Village\n4        M5A  Downtown Toronto                    Regent Park, Harbourfront\n5        M6A        North York             Lawrence Manor, Lawrence Heights\n6        M7A  Downtown Toronto  Queen's Park, Ontario Provincial Government"
                    },
                    "execution_count": 58,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "# More than one neighborhood can exist in one postal code area, separated by commas.\ndf['Neighborhood'] = (df['Neighborhood'].replace('\\/',',', regex=True)\n                        .replace(' ,',',', regex=True) # removes whitespace before comma/s\n                        .astype(str))\ndf.head()"
        },
        {
            "cell_type": "code",
            "execution_count": 59,
            "metadata": {},
            "outputs": [],
            "source": "# If a cell has a borough but a 'Not assigned' neighborhood, then the neighborhood will be the same as the borough.\ndf['Neighborhood'].replace('Not assigned', df['Borough'], inplace=True)"
        },
        {
            "cell_type": "code",
            "execution_count": 60,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/plain": "(103, 3)"
                    },
                    "execution_count": 60,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "# Use the .shape method to print the number of rows of your dataframe\ndf.shape"
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "## Step 2: Add geospatial data to the dataframe"
        },
        {
            "cell_type": "code",
            "execution_count": 61,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>PostalCode</th>\n      <th>Latitude</th>\n      <th>Longitude</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>M1B</td>\n      <td>43.806686</td>\n      <td>-79.194353</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>M1C</td>\n      <td>43.784535</td>\n      <td>-79.160497</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>M1E</td>\n      <td>43.763573</td>\n      <td>-79.188711</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>M1G</td>\n      <td>43.770992</td>\n      <td>-79.216917</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>M1H</td>\n      <td>43.773136</td>\n      <td>-79.239476</td>\n    </tr>\n  </tbody>\n</table>\n</div>",
                        "text/plain": "  PostalCode   Latitude  Longitude\n0        M1B  43.806686 -79.194353\n1        M1C  43.784535 -79.160497\n2        M1E  43.763573 -79.188711\n3        M1G  43.770992 -79.216917\n4        M1H  43.773136 -79.239476"
                    },
                    "execution_count": 61,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "# Read in geospatial data\ngeo_df = pd.read_csv('http://cocl.us/Geospatial_data')\n# Rename Postal Code column\ngeo_df = geo_df.rename(columns={'Postal Code': 'PostalCode'})\n\ngeo_df.head()"
        },
        {
            "cell_type": "code",
            "execution_count": 64,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>PostalCode</th>\n      <th>Borough</th>\n      <th>Neighborhood</th>\n      <th>Latitude</th>\n      <th>Longitude</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>M3A</td>\n      <td>North York</td>\n      <td>Parkwoods</td>\n      <td>43.753259</td>\n      <td>-79.329656</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>M4A</td>\n      <td>North York</td>\n      <td>Victoria Village</td>\n      <td>43.725882</td>\n      <td>-79.315572</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>M5A</td>\n      <td>Downtown Toronto</td>\n      <td>Regent Park, Harbourfront</td>\n      <td>43.654260</td>\n      <td>-79.360636</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>M6A</td>\n      <td>North York</td>\n      <td>Lawrence Manor, Lawrence Heights</td>\n      <td>43.718518</td>\n      <td>-79.464763</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>M7A</td>\n      <td>Downtown Toronto</td>\n      <td>Queen's Park, Ontario Provincial Government</td>\n      <td>43.662301</td>\n      <td>-79.389494</td>\n    </tr>\n  </tbody>\n</table>\n</div>",
                        "text/plain": "  PostalCode           Borough                                 Neighborhood  \\\n0        M3A        North York                                    Parkwoods   \n1        M4A        North York                             Victoria Village   \n2        M5A  Downtown Toronto                    Regent Park, Harbourfront   \n3        M6A        North York             Lawrence Manor, Lawrence Heights   \n4        M7A  Downtown Toronto  Queen's Park, Ontario Provincial Government   \n\n    Latitude  Longitude  \n0  43.753259 -79.329656  \n1  43.725882 -79.315572  \n2  43.654260 -79.360636  \n3  43.718518 -79.464763  \n4  43.662301 -79.389494  "
                    },
                    "execution_count": 64,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": "# Merge the dataframes on common key PostalCode\ndf_merged = pd.merge(df, geo_df, on='PostalCode')\ndf_merged.head()"
        }
    ],
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.6",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.6.9"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 1
}