Clustering Neighborhoods in Toronto 2.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "# Segmenting and Clustering Neighborhoods in Toronto", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "### Coursera Capstone Project 2 Part 2", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## 2. Loading the Dataframe from the previous Notebook", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 1, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [], 
            "source": "# installing and importing packages\nimport pandas as pd\nimport numpy as np"
        }, 
        {
            "source": "Now we can use the pandas library to load and read the dataframe 'df' that was created by scraping the raw data from the Wikipedia page: ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 2, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": false
            }, 
            "outputs": [
                {
                    "execution_count": 2, 
                    "metadata": {}, 
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>PostalCode</th>\n      <th>Borough</th>\n      <th>Neighborhood</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>M1B</td>\n      <td>Scarborough</td>\n      <td>Rouge, Malvern</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>M1C</td>\n      <td>Scarborough</td>\n      <td>Highland Creek, Rouge Hill, Port Union</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>M1E</td>\n      <td>Scarborough</td>\n      <td>Guildwood, Morningside, West Hill</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>M1G</td>\n      <td>Scarborough</td>\n      <td>Woburn</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>M1H</td>\n      <td>Scarborough</td>\n      <td>Cedarbrae</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>M1J</td>\n      <td>Scarborough</td>\n      <td>Scarborough Village</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>M1K</td>\n      <td>Scarborough</td>\n      <td>East Birchmount Park, Ionview, Kennedy Park</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>M1L</td>\n      <td>Scarborough</td>\n      <td>Clairlea, Golden Mile, Oakridge</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>M1M</td>\n      <td>Scarborough</td>\n      <td>Cliffcrest, Cliffside, Scarborough Village West</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>M1N</td>\n      <td>Scarborough</td>\n      <td>Birch Cliff, Cliffside West</td>\n    </tr>\n  </tbody>\n</table>\n</div>", 
                        "text/plain": "  PostalCode      Borough                                     Neighborhood\n0        M1B  Scarborough                                   Rouge, Malvern\n1        M1C  Scarborough           Highland Creek, Rouge Hill, Port Union\n2        M1E  Scarborough                Guildwood, Morningside, West Hill\n3        M1G  Scarborough                                           Woburn\n4        M1H  Scarborough                                        Cedarbrae\n5        M1J  Scarborough                              Scarborough Village\n6        M1K  Scarborough      East Birchmount Park, Ionview, Kennedy Park\n7        M1L  Scarborough                  Clairlea, Golden Mile, Oakridge\n8        M1M  Scarborough  Cliffcrest, Cliffside, Scarborough Village West\n9        M1N  Scarborough                      Birch Cliff, Cliffside West"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "df = pd.read_csv('Toronto Neighborhood Data.csv')\ndf.head(10)"
        }, 
        {
            "source": "Note that the code in the cell above will read the dataframe only after running the 'pd.to_csv()' command in the previous notebook.  ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## 3. Adding Geospatial Data to the Toronto Neighborhoods DataFrame", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Now, geocoder didn't work for me for retreiving coordinates for the neighborhoods. I tried a for loop, used arcgis and canadapost instead of google and even made a developer account and got the access key, but there seems to be something fundamentally wrong with the geocoder, so I'm gonna go ahead and use the csv file provided in the instructions:", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 3, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 3, 
                    "metadata": {}, 
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Postal Code</th>\n      <th>Latitude</th>\n      <th>Longitude</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>M1B</td>\n      <td>43.806686</td>\n      <td>-79.194353</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>M1C</td>\n      <td>43.784535</td>\n      <td>-79.160497</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>M1E</td>\n      <td>43.763573</td>\n      <td>-79.188711</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>M1G</td>\n      <td>43.770992</td>\n      <td>-79.216917</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>M1H</td>\n      <td>43.773136</td>\n      <td>-79.239476</td>\n    </tr>\n  </tbody>\n</table>\n</div>", 
                        "text/plain": "  Postal Code   Latitude  Longitude\n0         M1B  43.806686 -79.194353\n1         M1C  43.784535 -79.160497\n2         M1E  43.763573 -79.188711\n3         M1G  43.770992 -79.216917\n4         M1H  43.773136 -79.239476"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "df1 = pd.read_csv('https://cocl.us/Geospatial_data/Geospatial_Coordinates.csv')\ndf1.head()"
        }, 
        {
            "source": "The columns containing the postal codes in both the dataframes have slightly different names; the one in df1 has a space in between 'Postal' and 'Code'. This will cause an error while performing the join because Python wouldn't be able to match the column names. So let's remove the space by renaming the column in df1:", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 4, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [
                {
                    "execution_count": 4, 
                    "metadata": {}, 
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>PostalCode</th>\n      <th>Latitude</th>\n      <th>Longitude</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>M1B</td>\n      <td>43.806686</td>\n      <td>-79.194353</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>M1C</td>\n      <td>43.784535</td>\n      <td>-79.160497</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>M1E</td>\n      <td>43.763573</td>\n      <td>-79.188711</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>M1G</td>\n      <td>43.770992</td>\n      <td>-79.216917</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>M1H</td>\n      <td>43.773136</td>\n      <td>-79.239476</td>\n    </tr>\n  </tbody>\n</table>\n</div>", 
                        "text/plain": "  PostalCode   Latitude  Longitude\n0        M1B  43.806686 -79.194353\n1        M1C  43.784535 -79.160497\n2        M1E  43.763573 -79.188711\n3        M1G  43.770992 -79.216917\n4        M1H  43.773136 -79.239476"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "df1.rename(columns={'Postal Code':'PostalCode'}, inplace=True)\ndf1.head()"
        }, 
        {
            "source": "Now we can merge the two dataframes using the .join() method. We perform an outer join on the 'PostalCode' column to get the corresponding coordinates for the neighborhoods.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 5, 
                    "metadata": {}, 
                    "data": {
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>PostalCode</th>\n      <th>Borough</th>\n      <th>Neighborhood</th>\n      <th>Latitude</th>\n      <th>Longitude</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>M1B</td>\n      <td>Scarborough</td>\n      <td>Rouge, Malvern</td>\n      <td>43.806686</td>\n      <td>-79.194353</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>M1C</td>\n      <td>Scarborough</td>\n      <td>Highland Creek, Rouge Hill, Port Union</td>\n      <td>43.784535</td>\n      <td>-79.160497</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>M1E</td>\n      <td>Scarborough</td>\n      <td>Guildwood, Morningside, West Hill</td>\n      <td>43.763573</td>\n      <td>-79.188711</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>M1G</td>\n      <td>Scarborough</td>\n      <td>Woburn</td>\n      <td>43.770992</td>\n      <td>-79.216917</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>M1H</td>\n      <td>Scarborough</td>\n      <td>Cedarbrae</td>\n      <td>43.773136</td>\n      <td>-79.239476</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>M1J</td>\n      <td>Scarborough</td>\n      <td>Scarborough Village</td>\n      <td>43.744734</td>\n      <td>-79.239476</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>M1K</td>\n      <td>Scarborough</td>\n      <td>East Birchmount Park, Ionview, Kennedy Park</td>\n      <td>43.727929</td>\n      <td>-79.262029</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>M1L</td>\n      <td>Scarborough</td>\n      <td>Clairlea, Golden Mile, Oakridge</td>\n      <td>43.711112</td>\n      <td>-79.284577</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>M1M</td>\n      <td>Scarborough</td>\n      <td>Cliffcrest, Cliffside, Scarborough Village West</td>\n      <td>43.716316</td>\n      <td>-79.239476</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>M1N</td>\n      <td>Scarborough</td>\n      <td>Birch Cliff, Cliffside West</td>\n      <td>43.692657</td>\n      <td>-79.264848</td>\n    </tr>\n  </tbody>\n</table>\n</div>", 
                        "text/plain": "  PostalCode      Borough                                     Neighborhood  \\\n0        M1B  Scarborough                                   Rouge, Malvern   \n1        M1C  Scarborough           Highland Creek, Rouge Hill, Port Union   \n2        M1E  Scarborough                Guildwood, Morningside, West Hill   \n3        M1G  Scarborough                                           Woburn   \n4        M1H  Scarborough                                        Cedarbrae   \n5        M1J  Scarborough                              Scarborough Village   \n6        M1K  Scarborough      East Birchmount Park, Ionview, Kennedy Park   \n7        M1L  Scarborough                  Clairlea, Golden Mile, Oakridge   \n8        M1M  Scarborough  Cliffcrest, Cliffside, Scarborough Village West   \n9        M1N  Scarborough                      Birch Cliff, Cliffside West   \n\n    Latitude  Longitude  \n0  43.806686 -79.194353  \n1  43.784535 -79.160497  \n2  43.763573 -79.188711  \n3  43.770992 -79.216917  \n4  43.773136 -79.239476  \n5  43.744734 -79.239476  \n6  43.727929 -79.262029  \n7  43.711112 -79.284577  \n8  43.716316 -79.239476  \n9  43.692657 -79.264848  "
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "df = df.join(df1.set_index('PostalCode'), on='PostalCode', how='outer')\ndf.head(10)"
        }, 
        {
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 6, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "(103, 5)"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "# just checking the dimensions of the data to see if everything matches\ndf.shape"
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# saving this new dataframe as a csv file\ndf.to_csv('Toronto Geospatial Data.csv', index=False)"
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 3.5", 
            "name": "python3", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "3.5.5", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython3", 
            "codemirror_mode": {
                "version": 3, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}