data_summary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eefe0b4f-1243-4cfd-a144-342144cefdc1",
   "metadata": {},
   "source": [
    "# 2. Data Summary "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9acfefd-b289-4fe8-a18f-7a531815cd4b",
   "metadata": {},
   "source": [
    "## 2.1 Rearranging the data and plotting\n",
    "\n",
    "I decided that the most obvious and easiest way to analysis of this data was to visualise where and when the rides were taking place. To start, I worked with the data to create .csvs of the location by zone as defined in the provided shapefile and time by hour. These csvs have the location id as the row, and the hour as the column, including an additional row for all data. Each cell contains the total number of trips for each hour in each location. The code below produces two csvs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "569603c3-4fc9-4d3b-8448-41fdeb369a80",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import cartopy.io.shapereader as shpreader\n",
    "\n",
    "input = pd.read_csv('tlc_yellow_trips_2018_11_22_CLEAN.csv')\n",
    "print(input.head())\n",
    "\n",
    "input['pickup_datetime']=pd.to_datetime(input['pickup_datetime'])\n",
    "input['dropoff_datetime']=pd.to_datetime(input['dropoff_datetime'])\n",
    "\n",
    "shpfilename = '/home/Earth/mfalls/Downloads/junior-data-scientist-test-data-team-master/tlc_yellow_geom.shp'\n",
    "reader = shpreader.Reader(shpfilename)\n",
    "zones = reader.records()\n",
    "zone_ids = []\n",
    "for zone in zones:\n",
    "    zone_ids.append(int(zone.attributes['zone_id']))\n",
    "zone_ids = sorted(zone_ids)\n",
    "n = len(zone_ids)\n",
    "\n",
    "pickup = pd.DataFrame({'zone_id':zone_ids, 'all':np.zeros((n))})\n",
    "dropoff = pd.DataFrame({'zone_id':zone_ids, 'all':np.zeros((n))})\n",
    "pickup = pickup.set_index('zone_id')\n",
    "dropoff = dropoff.set_index('zone_id')\n",
    "\n",
    "for h in range(24):\n",
    "    pickup[str(h)] = np.zeros((n))\n",
    "    dropoff[str(h)] = np.zeros((n))\n",
    "for r in range(input.shape[0]):\n",
    "    puID = input['pickup_location_id'][r]\n",
    "    doID = input['dropoff_location_id'][r]\n",
    "    puH = input['pickup_datetime'][r].hour\n",
    "    doH = input['dropoff_datetime'][r].hour\n",
    "    pickup['all'][puID] = pickup['all'][puID] + 1\n",
    "    pickup[str(puH)][puID] = pickup[str(puH)][puID] + 1\n",
    "    dropoff['all'][doID] = dropoff['all'][doID] + 1\n",
    "    dropoff[str(doH)][doID] = dropoff[str(doH)][doID] + 1\n",
    "\n",
    "pickup.to_csv('pickup_timeplace.csv')\n",
    "dropoff.to_csv('dropoff_timeplace.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72e1b6e7-9912-4ab9-9bcd-f2b187a94da0",
   "metadata": {},
   "source": [
    "The code below plots the hourly data, and presents hourly data in subplots for both pickup and dropoff. To make comparison easier, pickup and dropoff use the same scale. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20e96cc2-67da-44a9-9eef-2f22dcf6370b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import cartopy.io.shapereader as shpreader\n",
    "import cartopy.crs as ccrs\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from shapely.geometry import MultiPolygon\n",
    "import math as m\n",
    "\n",
    "pickup = pd.read_csv('pickup_timeplace.csv')\n",
    "dropoff = pd.read_csv('dropoff_timeplace.csv')\n",
    "zone_ids = np.array(pickup['zone_id'])\n",
    "pickup = pickup.set_index('zone_id')\n",
    "dropoff = dropoff.set_index('zone_id')\n",
    "pickmax = np.max(pickup.to_numpy()[:,1:])\n",
    "dropmax = np.max(dropoff.to_numpy()[:,1:])\n",
    "hourlymax = max([pickmax,dropmax])\n",
    "\n",
    "datas = [pickup,dropoff]\n",
    "names = ['pickup','dropoff']\n",
    "\n",
    "for i in range(2):\n",
    "    data = datas[i]\n",
    "    name = names[i]\n",
    "    fig, ax = plt.subplots(4,6,figsize=[12.8,9.6])\n",
    "    fig.tight_layout()\n",
    "    for h in range(24):\n",
    "        j=m.floor(h/6)\n",
    "        i=h%6\n",
    "        ax = plt.subplot(4, 6, h+1, projection=ccrs.PlateCarree())\n",
    "        ax.set_extent([-74.27, -73.68, 40.48, 40.92], ccrs.PlateCarree())\n",
    "        shpfilename = '/home/Earth/mfalls/Downloads/junior-data-scientist-test-data-team-master/tlc_yellow_geom.shp'\n",
    "        reader = shpreader.Reader(shpfilename)\n",
    "        zones = reader.records()\n",
    "        for zone in zones:\n",
    "            zid = zone.attributes['zone_id']\n",
    "            if zid not in ['56', '103']:\n",
    "                r = data[str(h)][int(zid)]\n",
    "                try:\n",
    "                    if r == 0:\n",
    "                        ax.add_geometries(zone.geometry, ccrs.PlateCarree(), facecolor='#FFFFFF', linewidth=0.1,\n",
    "                                          edgecolor=\"black\")\n",
    "                    elif 0 < r and r <= 100:\n",
    "                        ax.add_geometries(zone.geometry, ccrs.PlateCarree(), facecolor='#FFECEC', linewidth=0.1,\n",
    "                                          edgecolor=\"black\")\n",
    "                    elif 100 < r and r <= 200:\n",
    "                        ax.add_geometries(zone.geometry, ccrs.PlateCarree(), facecolor='#FFB5B5', linewidth=0.1,\n",
    "                                          edgecolor=\"black\")\n",
    "                    elif 200 < r and r <= 300:\n",
    "                        ax.add_geometries(zone.geometry, ccrs.PlateCarree(), facecolor='#ff7575', linewidth=0.1,\n",
    "                                          edgecolor=\"black\")\n",
    "                    elif 300 < r and r <= 400:\n",
    "                        ax.add_geometries(zone.geometry, ccrs.PlateCarree(), facecolor='#FF2D2D', linewidth=0.1,\n",
    "                                          edgecolor=\"black\")\n",
    "                    elif 400 < r and r <= 500:\n",
    "                        ax.add_geometries(zone.geometry, ccrs.PlateCarree(), facecolor='#EA0000', linewidth=0.1,\n",
    "                                          edgecolor=\"black\")\n",
    "                    elif  r > 500:\n",
    "                        ax.add_geometries(zone.geometry, ccrs.PlateCarree(), facecolor='#AE0000', linewidth=0.1,\n",
    "                                          edgecolor=\"black\")\n",
    "                except Exception as e:\n",
    "                    list_str_polygons = [str(zone.geometry)]\n",
    "                    c = MultiPolygon(map(wkt.loads, list_str_polygons))\n",
    "                    if r == 0:\n",
    "                        ax.add_geometries(c, ccrs.PlateCarree(), facecolor='#FFFFFF', linewidth=0.1, edgecolor=\"black\")\n",
    "                    elif 0 < r and r <= 100:\n",
    "                        ax.add_geometries(c, ccrs.PlateCarree(), facecolor='#FFECEC', linewidth=0.1, edgecolor=\"black\")\n",
    "                    elif 100 < r and r <= 200:\n",
    "                        ax.add_geometries(c, ccrs.PlateCarree(), facecolor='#FFB5B5', linewidth=0.1, edgecolor=\"black\")\n",
    "                    elif 200 < r and r <= 300:\n",
    "                        ax.add_geometries(c, ccrs.PlateCarree(), facecolor='#ff7575', linewidth=0.1, edgecolor=\"black\")\n",
    "                    elif 300 < r and r <= 400:\n",
    "                        ax.add_geometries(c, ccrs.PlateCarree(), facecolor='#FF2D2D', linewidth=0.1, edgecolor=\"black\")\n",
    "                    elif 400 < r and r <= 500:\n",
    "                        ax.add_geometries(c, ccrs.PlateCarree(), facecolor='#EA0000', linewidth=0.1, edgecolor=\"black\")\n",
    "                    elif r > 500:\n",
    "                        ax.add_geometries(c, ccrs.PlateCarree(), facecolor='#AE0000', linewidth=0.1, edgecolor=\"black\")\n",
    "\n",
    "        ax.set_title(str(h)+ '.00-' + str(h+1) + '.00')\n",
    "\n",
    "    print(name + '_locs_hourly.png')\n",
    "    plt.show()\n",
    "    plt.savefig(name + '_locs_hourly.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b9c99b9-0f68-4b44-81b2-b1c64008009b",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2.2 Commentary on Data\n",
    "\n",
    "Looking at both the pickup and dropoff data, we notice that the most frequent area for both pickup and dropoff is in the midtown area of Manhattan, starting from around noon and continuing until late evening. When we compare this to [NYC population density data](https://viewing.nyc/media/ac2b6e5c581107f23f1a3648986719f1/), we can see that this isn't the most densily populated area in the city, and given that it's a holiday, not many people will be going to work there. So why is this location the dominant trend? Considering events of the day, we can see that the region most in red corresponds to the time and route of [Macy's Thankgiving Parade](https://www.6sqft.com/map-where-to-watch-the-macys-thanksgiving-day-parade/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}