assets/books/cnbr/rg_cnbr_15_11_2019.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bangalore: The Battle of Neighborhoods\n",
    "\n",
    "Last Updated: Nov 15, 2019\n",
    "\n",
    "- Building a dataframe of neighborhoods in Bangalore, Karnataka by web scraping from Wikipedia\n",
    "- Getting the geographical coordinates of neighborhoods\n",
    "- Obtaining the venue data for neighborhoods from Foursquare API\n",
    "- Exploring and clustering the neighborhoods\n",
    "\n",
    "***\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Acknowledgement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project is a result of inspiration taken from Chia Lim's work on opening a Shopping Mall in Kuala Lumpur, Malaysia which can be accessed [here](https://github.com/limchiahooi/Coursera_Capstone/).\n",
    "\n",
    "It's possible that you might end up getting different results than those posted on my blog (depending on Foursquare data) but you should still see a similar trend among 6 clusters especially at the peripheries of the map (with discrepancies in the centre of the city - Central zone divided either into North & South or East & West)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'd also like to thank the IBM teaching staff for providing the required skills and tools to complete this project. This is an updated version of my submission in the last week of October, 2019. To use this notebook, you need to create a Foursquare API account [here](https://developer.foursquare.com/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# library to handle data in a vectorized manner\n",
    "import numpy as np \n",
    "\n",
    "# library for data analysis\n",
    "import pandas as pd \n",
    "pd.set_option(\"display.max_columns\", None)\n",
    "pd.set_option(\"display.max_rows\", None)\n",
    "\n",
    "# library to handle JSON files\n",
    "import json \n",
    "\n",
    "# installing geocoder and converting addresses into latitude and longitude values\n",
    "!pip install geocoder\n",
    "import geocoder\n",
    "\n",
    "# package for getting coordinates\n",
    "!pip install geopy\n",
    "from geopy.geocoders import Nominatim\n",
    "\n",
    "# library to handle requests\n",
    "import requests \n",
    "\n",
    "# library to parse HTML and XML documents\n",
    "!pip install beautifulsoup4\n",
    "from bs4 import BeautifulSoup \n",
    "\n",
    "# tranforming JSON file into a pandas dataframe\n",
    "from pandas.io.json import json_normalize \n",
    "\n",
    "# plotting modules: Matplotlib and MPL\n",
    "import matplotlib.cm as cm\n",
    "import matplotlib.colors as colors\n",
    "from matplotlib.pyplot import figure\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D \n",
    "\n",
    "# importing preprocessing tools to scale features\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# importing k-means from clustering stage\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "# importing clustering visualizer\n",
    "!pip install yellowbrick\n",
    "from yellowbrick.cluster import KElbowVisualizer\n",
    "\n",
    "# map rendering library\n",
    "!pip install folium\n",
    "import folium \n",
    "\n",
    "# importing package and its set of stopwords\n",
    "!pip install wordcloud\n",
    "from wordcloud import WordCloud, STOPWORDS\n",
    "\n",
    "print(\"Libraries imported.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scrapping data from Wikipedia into DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sending the GET request\n",
    "data = requests.get(\"https://en.wikipedia.org/wiki/Category:Neighbourhoods_in_Bangalore\").text\n",
    "\n",
    "# parsing data from the html into a beautifulsoup object\n",
    "soup = BeautifulSoup(data, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating a list to store neighborhood data\n",
    "neighborhoodList = []\n",
    "\n",
    "# appending the data into the list\n",
    "for row in soup.find_all(\"div\", class_=\"mw-category\")[0].findAll(\"li\"):\n",
    "    neighborhoodList.append(row.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating a new DataFrame from the list\n",
    "bl_df = pd.DataFrame({\"Neighborhood\": neighborhoodList})\n",
    "bl_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dropping the first three rows since they don't contain neighborhood data\n",
    "bl_df.drop(bl_df.index[0:3], inplace=True)\n",
    "bl_df.reset_index(inplace=True)\n",
    "bl_df.drop([\"index\"],axis=1,inplace=True)\n",
    "bl_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# printing the number of rows of the dataframe\n",
    "print(\"Rows in DataFrame [Number of Neigborhoods in Bangalore]:\", bl_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting the geographical coordinates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining a function to get coordinates, initializing and looping to get the coordinates\n",
    "def get_ll(neighborhood):\n",
    "    llc = None\n",
    "    while(llc is None):\n",
    "        str_neigh = neighborhood + ', Bengaluru, Karnataka'\n",
    "        g = geocoder.arcgis(str_neigh)\n",
    "        llc = g.latlng\n",
    "        # print(llc)\n",
    "    return llc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# calling the function to get coordinates and storing in a new list using list comprehension\n",
    "coords_l = []\n",
    "\n",
    "for neighborhood in bl_df[\"Neighborhood\"].tolist():\n",
    "    coords = get_ll(neighborhood)\n",
    "    coords_l.append(coords) \n",
    "\n",
    "print(\"Coordinates of neighborhoods:\", coords_l[0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating temporary dataframe to populate coordinates into Latitude and Longitude\n",
    "df_coords = pd.DataFrame(coords_l, columns=['Latitude', 'Longitude'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# merging the coordinates into the original dataframe\n",
    "bl_df['Latitude'] = df_coords['Latitude']\n",
    "bl_df['Longitude'] = df_coords['Longitude']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# checking the neighborhoods and coordinates\n",
    "bl_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving the DataFrame as CSV file\n",
    "bl_df.to_csv(\"bl_df.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a map of Bangalore with mapped neighborhoods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# getting the coordinates of Bengaluru\n",
    "address = 'Bengaluru, Karnataka'\n",
    "\n",
    "geolocator = Nominatim(user_agent=\"my-application\")\n",
    "location = geolocator.geocode(address)\n",
    "latitude = location.latitude\n",
    "longitude = location.longitude\n",
    "print('The geograpical coordinates of Bengaluru, Karnataka are {}, {}.'.format(latitude, longitude))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating a map of Bangalore using latitude and longitude values\n",
    "map_bl = folium.Map(location=[latitude, longitude], zoom_start=10.5)\n",
    "\n",
    "# add markers to map\n",
    "for lat, lng, neighborhood in zip(bl_df['Latitude'], bl_df['Longitude'], bl_df['Neighborhood']):\n",
    "    label = '{}'.format(neighborhood)\n",
    "    label = folium.Popup(label, parse_html=True)\n",
    "    folium.CircleMarker(\n",
    "        [lat, lng],\n",
    "        radius=2,\n",
    "        popup=label,\n",
    "        color='blue',\n",
    "        fill=True,\n",
    "        fill_color='#3186cc',\n",
    "        fill_opacity=0.7).add_to(map_bl)  \n",
    "    \n",
    "map_bl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Foursquare API to explore neighborhoods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining Foursquare Credentials and Version including ID, Secret & API\n",
    "CLIENT_ID = 'Enter your ID' \n",
    "CLIENT_SECRET = 'Enter your Secret' \n",
    "VERSION = '20180605'\n",
    "\n",
    "print('Your credentails:\\n')\n",
    "print('CLIENT_ID: ' + CLIENT_ID)\n",
    "print('CLIENT_SECRET: ' + CLIENT_SECRET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Getting top 100 venues in a 2 km radius of each neighborhood**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calling Foursquare API\n",
    "radius = 2000\n",
    "LIMIT = 100\n",
    "\n",
    "venues = []\n",
    "\n",
    "for lat, long, neighborhood in zip(bl_df['Latitude'], bl_df['Longitude'], bl_df['Neighborhood']):\n",
    "    \n",
    "    # creating the API request URL\n",
    "    url = \"https://api.foursquare.com/v2/venues/explore?client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}\".format(\n",
    "        CLIENT_ID,\n",
    "        CLIENT_SECRET,\n",
    "        VERSION,\n",
    "        lat,\n",
    "        long,\n",
    "        radius, \n",
    "        LIMIT)\n",
    "    \n",
    "    # making the GET request\n",
    "    results_mined = requests.get(url).json()\n",
    "    \n",
    "    results = results_mined['response']['groups'][0]['items']\n",
    "    \n",
    "    # returning only relevant information for each nearby venue\n",
    "    for venue in results:\n",
    "        venues.append((\n",
    "            neighborhood,\n",
    "            lat, \n",
    "            long, \n",
    "            venue['venue']['name'], \n",
    "            venue['venue']['location']['lat'], \n",
    "            venue['venue']['location']['lng'],  \n",
    "            venue['venue']['categories'][0]['name']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting the venues list into a new DataFrame\n",
    "venues_df = pd.DataFrame(venues)\n",
    "\n",
    "# defining column names\n",
    "venues_df.columns = ['Neighborhood', 'Latitude', 'Longitude', 'VenueName', 'VenueLatitude', 'VenueLongitude', 'VenueCategory']\n",
    "\n",
    "venues_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sorting the venues DataFrame\n",
    "venues_df.sort_values(by=['VenueLongitude'])\n",
    "\n",
    "venues_df_l_l = venues_df[[\"Neighborhood\",\"Latitude\",\"Longitude\"]]\n",
    "venues_df_g = venues_df_l_l.groupby(\"Neighborhood\").mean()\n",
    "\n",
    "lati_data = venues_df_g[\"Latitude\"]\n",
    "long_data = venues_df_g[\"Longitude\"]\n",
    "\n",
    "venues_df_g.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# checking for duplicates using a loop\n",
    "dupl_c=0\n",
    "n=len(venues_df['Neighborhood'].values)\n",
    "bool_series = venues_df.duplicated(subset=None, keep='first')\n",
    "\n",
    "for i in range(n):\n",
    "    if bool_series[i]==True:\n",
    "        dupl_c=dupl_c+1\n",
    "\n",
    "if dupl_c==0:\n",
    "    print(\"No duplicates\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# counting various venues after grouping by neighborhood\n",
    "venues_df_count = venues_df.groupby([\"Neighborhood\"]).count()\n",
    "print(\"Count of venues for each neighborhood:\",\"\\n\\n\", np.asanyarray(venues_df_count[\"VenueName\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Checking if neighborhoods extracted from Wiki match those obtained from venue data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# comparing values of sets\n",
    "neigh_wiki=bl_df[\"Neighborhood\"].tolist()\n",
    "neigh_fsq=venues_df[\"Neighborhood\"].tolist()\n",
    "\n",
    "print(\"Neighborhood not scanned:\", (set(neigh_wiki)-set(neigh_fsq)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# storing out the list of categories\n",
    "unique_categ = venues_df['VenueCategory'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wordcloud to check for popular places"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating DataFrame of venue categories\n",
    "newTest_words = venues_df[['VenueCategory']]\n",
    "newTest_words.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# writing categories to file and creating a cloud object\n",
    "newTest_words.to_csv('blr_venue_categ.txt', sep=',', index=False)\n",
    "\n",
    "myTest = open('blr_venue_categ.txt', 'r').read()\n",
    "stopwords = set(STOPWORDS)\n",
    "\n",
    "bl_v = WordCloud(\n",
    "background_color='white',\n",
    "    max_words=2000,\n",
    "    stopwords=stopwords)\n",
    "\n",
    "bl_v.generate(myTest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# displaying the word cloud\n",
    "fig = plt.figure()\n",
    "\n",
    "# setting width\n",
    "fig.set_figwidth(15) \n",
    "\n",
    "# setting height\n",
    "fig.set_figheight(20) \n",
    "\n",
    "plt.imshow(bl_v, interpolation='bilinear')\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One hot encoding of Venue Categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# one hot encoding\n",
    "bl_onehot = pd.get_dummies(venues_df[['VenueCategory']], prefix=\"\", prefix_sep=\"\")\n",
    "\n",
    "# adding neighborhood column back to dataframe\n",
    "bl_onehot['Neighborhoods'] = venues_df['Neighborhood'] \n",
    "\n",
    "# moving neighborhood column to the first column\n",
    "fixed_columns = [bl_onehot.columns[-1]] + list(bl_onehot.columns[:-1])\n",
    "bl_onehot = bl_onehot[fixed_columns]\n",
    "\n",
    "bl_onehot.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# grouping venues of each category by neighborhood\n",
    "bl_grouped = bl_onehot.groupby([\"Neighborhoods\"]).sum()\n",
    "bl_grouped.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Getting total number of venues classified as restaurants for each neighborhood**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting a string keyword to go through the venue titles and appending to DataFrame\n",
    "str1 = 'Restaurant'\n",
    "col_list = []\n",
    "\n",
    "for column_id in bl_grouped.columns:\n",
    "    if str1 in column_id:\n",
    "        col_list.append(column_id)\n",
    "        \n",
    "# adding up count of restaurants in all neighborhoods \n",
    "t_r_f = 0\n",
    "ven_val = venues_df.VenueCategory\n",
    "\n",
    "for v_c in range(len(ven_val)):\n",
    "    if str1 in ven_val[v_c]:\n",
    "        t_r_f = t_r_f + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# printing types of restaurants after validation\n",
    "bl_grouped['Total Restaurants'] = bl_grouped[col_list].sum(axis=1)\n",
    "\n",
    "t_r_cc = np.asanyarray(bl_grouped['Total Restaurants'])\n",
    "\n",
    "print(\"Total number of eateries categorized as restaurants:\", t_r_f, \"\\n\")  \n",
    "if t_r_f == t_r_cc.sum():\n",
    "    print(\"Restaurant division classification has been verified\",\"\\n\") \n",
    "print(\"Types of Restaurants:\", col_list)\n",
    "\n",
    "bl_grouped['Latitude'] = lati_data\n",
    "bl_grouped['Longitude'] = long_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# resetting indexes\n",
    "bl_grouped = bl_grouped.reset_index()\n",
    "bl_grouped.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extracting and grouping all venues classified as restaurants for each neighborhood\n",
    "bl_r_1 = bl_grouped[[\"Neighborhoods\",\"Total Restaurants\",\"Latitude\",\"Longitude\"]]\n",
    "bl_r_1.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering of restaurants by neighborhood"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preprocessing for scaling\n",
    "bl_clustering = bl_r_1.drop([\"Neighborhoods\"], 1)\n",
    "\n",
    "# feature scaling\n",
    "X = bl_clustering\n",
    "X = np.nan_to_num(X)\n",
    "Clus_dataSet = MinMaxScaler().fit_transform(X)\n",
    "Clus_dataSet[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# finding the elbow region for optimal cluster size\n",
    "krng = np.arange(2, 15)\n",
    "sse = []\n",
    "\n",
    "for k in krng:\n",
    "    km = KMeans(init = \"k-means++\", n_clusters = k, n_init = 200, max_iter = 400)\n",
    "    km.fit(Clus_dataSet)\n",
    "    sse.append(km.inertia_)\n",
    "    \n",
    "plt.figure(1, figsize=(6, 4), dpi=100, facecolor='w', edgecolor='k')\n",
    "\n",
    "plt.scatter(krng, sse)\n",
    "plt.plot(krng, sse)\n",
    "\n",
    "frame_plt = plt.gca()\n",
    "\n",
    "frame_plt.axes.get_yaxis().set_ticks([])\n",
    "\n",
    "plt.xlabel('Number of Clusters', fontsize=12)\n",
    "plt.ylabel('Inertia', fontsize=12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting number of clusters\n",
    "kclusters = 6\n",
    "\n",
    "# running k-means clustering\n",
    "kmeans = KMeans(init = \"k-means++\", n_clusters = kclusters, n_init = 200, max_iter = 400).fit(Clus_dataSet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating a new dataframe that includes cluster labels\n",
    "bl_merged = bl_r_1.copy()\n",
    "bl_merged[\"Labels\"] = kmeans.labels_\n",
    "\n",
    "# displaying cluster centers\n",
    "kmeans.cluster_centers_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# renaming columns for merging\n",
    "bl_merged.rename(columns={\"Neighborhoods\": \"Neighborhood\"}, inplace=True)\n",
    "bl_merged.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sorting results by Cluster Labels\n",
    "bl_merged.sort_values([\"Labels\"], inplace=True)\n",
    "bl_merged = bl_merged[[\"Neighborhood\",\"Total Restaurants\",\n",
    "                       \"Labels\", \"Latitude\",\"Longitude\"]]\n",
    "\n",
    "t_r_c = np.asanyarray(bl_merged[\"Total Restaurants\"])\n",
    "c_c = []\n",
    "\n",
    "for i in range(bl_merged.shape[0]):\n",
    "    if t_r_c[i] <40:\n",
    "        if t_r_c[i] <20:\n",
    "            c_c.append(\"Low\")\n",
    "        else:\n",
    "            c_c.append(\"Medium\")\n",
    "    else:\n",
    "        c_c.append(\"High\")\n",
    "        \n",
    "bl_merged[\"Restaurant Density\"] = c_c\n",
    "\n",
    "# setting index name\n",
    "bl_merged.index.name=\"Index Number\"\n",
    "bl_merged.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Counting occurrence of restaurant densities\n",
    "rest_density=pd.get_dummies(bl_merged[[\"Restaurant Density\"]], prefix=\"\", prefix_sep=\"\").sum()\n",
    "rest_dens_c = rest_density.to_frame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Renaming columns for display purposes\n",
    "rest_dens_c.reset_index(inplace=True)\n",
    "rest_dens_c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tabulating for visualization\n",
    "col_n_u = ['Restaurant Density','Count']\n",
    "rest_dens_c.columns = col_n_u\n",
    "rest_dens_c[\"Level\"] = [2,0,1]\n",
    "rest_dens_c = rest_dens_c.sort_values(\"Level\").drop(\"Level\",axis=1)\n",
    "rest_dens_c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualization of restaurant density categories\n",
    "plt.figure(1, figsize=(6, 4), dpi=100, facecolor='w', edgecolor='k')\n",
    "plt.bar(rest_dens_c[\"Restaurant Density\"], rest_dens_c[\"Count\"])\n",
    "\n",
    "plt.ylabel(\"Count\", fontsize=12)\n",
    "plt.xlabel(\"Category\", fontsize=12)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization of Clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Scatter plot of restaurants grouped by cluster**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scatter plot in 3D to visualize clusters\n",
    "fig = plt.figure(1, figsize=(16, 12), dpi=200, facecolor='w', edgecolor='k')\n",
    "\n",
    "plt.clf()\n",
    "ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=48, azim=134)\n",
    "\n",
    "labels = bl_merged['Labels']\n",
    "\n",
    "ar = np.arange(1,128)\n",
    "area = np.pi * ( ar/(ar.max()))**2*150\n",
    "colors_td = cm.rainbow(labels.astype(np.float)/kclusters)\n",
    "\n",
    "plt.cla()\n",
    "\n",
    "ax.set_xlabel('Latitude', fontsize=16)\n",
    "ax.set_ylabel('Longitude', fontsize=16)\n",
    "ax.set_zlabel('Total Restaurants', fontsize=16)\n",
    "\n",
    "# Turn off tick labels\n",
    "ax.set_xticklabels([])\n",
    "ax.set_yticklabels([])\n",
    "ax.set_zticklabels([])\n",
    "\n",
    "pl_X=np.asanyarray(bl_merged[[\"Latitude\",\"Longitude\",\"Total Restaurants\"]])\n",
    "\n",
    "ax.scatter(pl_X[:, 0], pl_X[:, 1], pl_X[:, 2], s=area, c=colors_td)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Folium plot of restaurants grouped by cluster**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating map\n",
    "map_clusters = folium.Map(location=[latitude, longitude], zoom_start=10.5)\n",
    "\n",
    "# setting color scheme for clusters\n",
    "x = np.arange(kclusters)\n",
    "ys = [i+x+(i*x)**3 for i in range(kclusters)]\n",
    "colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))\n",
    "rainbow = [colors.rgb2hex(i) for i in colors_array]\n",
    "\n",
    "# adding markers to map\n",
    "markers_colors = []\n",
    "for lat, lon, poi, cluster in zip(bl_merged['Latitude'], bl_merged['Longitude'], bl_merged['Neighborhood'], bl_merged['Labels']):\n",
    "    label = folium.Popup(str(poi) + ' - Cluster ' + str(cluster), parse_html=True)\n",
    "    folium.CircleMarker(\n",
    "        [lat, lon],\n",
    "        radius=5,\n",
    "        popup=label,\n",
    "        color=\"black\",\n",
    "        weight=1,\n",
    "        fill=True,\n",
    "        fill_color=rainbow[cluster],\n",
    "        fill_opacity=2).add_to(map_clusters)\n",
    "       \n",
    "map_clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cluster Information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cluster 0**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bl_r_clus_1 = bl_merged[bl_merged.Labels==0]\n",
    "bl_r_clus_1.sort_values([\"Total Restaurants\"], inplace=True)\n",
    "        \n",
    "bl_r_clus_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cluster 1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bl_r_clus_2 = bl_merged[bl_merged.Labels==1]\n",
    "bl_r_clus_2.sort_values([\"Total Restaurants\"], inplace=True)\n",
    "\n",
    "bl_r_clus_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cluster 2**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bl_r_clus_3 = bl_merged[bl_merged.Labels==2]\n",
    "bl_r_clus_3.sort_values([\"Total Restaurants\"], inplace=True)\n",
    "\n",
    "bl_r_clus_3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cluster 3**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bl_r_clus_4 = bl_merged[bl_merged.Labels==3]\n",
    "bl_r_clus_4.sort_values([\"Total Restaurants\"], inplace=True)\n",
    "\n",
    "bl_r_clus_4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cluster 4**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bl_r_clus_5 = bl_merged[bl_merged.Labels==4]\n",
    "bl_r_clus_5.sort_values([\"Total Restaurants\"], inplace=True)\n",
    "\n",
    "bl_r_clus_5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cluster 5**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bl_r_clus_6 = bl_merged[bl_merged.Labels==5]\n",
    "bl_r_clus_6.sort_values([\"Total Restaurants\"], inplace=True)\n",
    "\n",
    "bl_r_clus_6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations\n",
    "\n",
    "There is a high demand for varied cuisine in the city of Bangalore and this is confirmed by Foursquare data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}