using-API-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Investigating crashes using API data\n",
    "A number of factors can increase the chances of a car crash occuring - for instance: weather, drink-driving or time of day. In this in project, I set out to collect data from the web in order to understand the circumstances surrounding car crashes in the state of Maryland, U.S. The crash data was taken from the U.S. website: data.gov. \n",
    "\n",
    "1. First of all, the Foursquare API is used to collect data about the number of bars within in a certain radius in each County, in order to see if the prevalence of drinking might have an influence on the accident rate. \n",
    "2. Second of all, the Google Maps API is used to determine the coordinates of each county, then the coordinates are used to request the weather conditions at the time of the accident from the DarkSky API.\n",
    "3. Finally, the coordinates of the accidents themselves are obtained and plotted on a map using the Google Map Plotter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "The data for this exercise can be found [here](https://catalog.data.gov/dataset/2012-vehicle-collisions-investigated-by-state-police-4fcd0/resource/d84f79b6-419c-49e0-a74c-01b34a9575f2).\n",
    "\n",
    "Just run the cells below to get the data ready."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "mypath = \"./\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(mypath + \"2012_Vehicle_Collisions_Investigated_by_State_Police.csv\",\n",
    "                   parse_dates=[[\"ACC_DATE\", \"ACC_TIME\"]])\n",
    "data[\"MONTH\"] = data.ACC_DATE_ACC_TIME.dt.month\n",
    "data.dropna(subset=[\"COUNTY_NAME\"], inplace=True) #get rid of empty counties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's check the length of the data and the names of the counties and the dataset itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18604"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Montgomery', 'Worcester', 'Calvert', 'St. Marys', 'Baltimore',\n",
       "       'Prince Georges', 'Anne Arundel', 'Cecil', 'Charles', 'Carroll',\n",
       "       'Harford', 'Frederick', 'Howard', 'Allegany', 'Garrett', 'Kent',\n",
       "       'Queen Annes', 'Washington', 'Somerset', 'Wicomico', 'Talbot',\n",
       "       'Caroline', 'Dorchester', 'Not Applicable', 'Unknown',\n",
       "       'Baltimore City'], dtype=object)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.COUNTY_NAME.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ACC_DATE_ACC_TIME</th>\n",
       "      <th>CASE_NUMBER</th>\n",
       "      <th>BARRACK</th>\n",
       "      <th>ACC_TIME_CODE</th>\n",
       "      <th>DAY_OF_WEEK</th>\n",
       "      <th>ROAD</th>\n",
       "      <th>INTERSECT_ROAD</th>\n",
       "      <th>DIST_FROM_INTERSECT</th>\n",
       "      <th>DIST_DIRECTION</th>\n",
       "      <th>CITY_NAME</th>\n",
       "      <th>COUNTY_CODE</th>\n",
       "      <th>COUNTY_NAME</th>\n",
       "      <th>VEHICLE_COUNT</th>\n",
       "      <th>PROP_DEST</th>\n",
       "      <th>INJURY</th>\n",
       "      <th>COLLISION_WITH_1</th>\n",
       "      <th>COLLISION_WITH_2</th>\n",
       "      <th>MONTH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 02:01:00</td>\n",
       "      <td>1363000002</td>\n",
       "      <td>Rockville</td>\n",
       "      <td>1</td>\n",
       "      <td>SUNDAY</td>\n",
       "      <td>IS 00495 CAPITAL BELTWAY</td>\n",
       "      <td>IS 00270 EISENHOWER MEMORIAL</td>\n",
       "      <td>0.00</td>\n",
       "      <td>U</td>\n",
       "      <td>Not Applicable</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Montgomery</td>\n",
       "      <td>2.0</td>\n",
       "      <td>YES</td>\n",
       "      <td>NO</td>\n",
       "      <td>VEH</td>\n",
       "      <td>OTHER-COLLISION</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 18:01:00</td>\n",
       "      <td>1296000023</td>\n",
       "      <td>Berlin</td>\n",
       "      <td>5</td>\n",
       "      <td>SUNDAY</td>\n",
       "      <td>MD 00090 OCEAN CITY EXPWY</td>\n",
       "      <td>CO 00220 ST MARTINS NECK RD</td>\n",
       "      <td>0.25</td>\n",
       "      <td>W</td>\n",
       "      <td>Not Applicable</td>\n",
       "      <td>23.0</td>\n",
       "      <td>Worcester</td>\n",
       "      <td>1.0</td>\n",
       "      <td>YES</td>\n",
       "      <td>NO</td>\n",
       "      <td>FIXED OBJ</td>\n",
       "      <td>OTHER-COLLISION</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 07:01:00</td>\n",
       "      <td>1283000016</td>\n",
       "      <td>Prince Frederick</td>\n",
       "      <td>2</td>\n",
       "      <td>SUNDAY</td>\n",
       "      <td>MD 00765 MAIN ST</td>\n",
       "      <td>CO 00208 DUKE ST</td>\n",
       "      <td>100.00</td>\n",
       "      <td>S</td>\n",
       "      <td>Not Applicable</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Calvert</td>\n",
       "      <td>1.0</td>\n",
       "      <td>YES</td>\n",
       "      <td>NO</td>\n",
       "      <td>FIXED OBJ</td>\n",
       "      <td>FIXED OBJ</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 00:01:00</td>\n",
       "      <td>1282000006</td>\n",
       "      <td>Leonardtown</td>\n",
       "      <td>1</td>\n",
       "      <td>SUNDAY</td>\n",
       "      <td>MD 00944 MERVELL DEAN RD</td>\n",
       "      <td>MD 00235 THREE NOTCH RD</td>\n",
       "      <td>10.00</td>\n",
       "      <td>E</td>\n",
       "      <td>Not Applicable</td>\n",
       "      <td>18.0</td>\n",
       "      <td>St. Marys</td>\n",
       "      <td>1.0</td>\n",
       "      <td>YES</td>\n",
       "      <td>NO</td>\n",
       "      <td>FIXED OBJ</td>\n",
       "      <td>OTHER-COLLISION</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 01:01:00</td>\n",
       "      <td>1267000007</td>\n",
       "      <td>Essex</td>\n",
       "      <td>1</td>\n",
       "      <td>SUNDAY</td>\n",
       "      <td>IS 00695 BALTO BELTWAY</td>\n",
       "      <td>IS 00083 HARRISBURG EXPWY</td>\n",
       "      <td>100.00</td>\n",
       "      <td>S</td>\n",
       "      <td>Not Applicable</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Baltimore</td>\n",
       "      <td>2.0</td>\n",
       "      <td>YES</td>\n",
       "      <td>NO</td>\n",
       "      <td>VEH</td>\n",
       "      <td>OTHER-COLLISION</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    ACC_DATE_ACC_TIME  CASE_NUMBER           BARRACK  ACC_TIME_CODE  \\\n",
       "0 2012-01-01 02:01:00   1363000002         Rockville              1   \n",
       "1 2012-01-01 18:01:00   1296000023            Berlin              5   \n",
       "2 2012-01-01 07:01:00   1283000016  Prince Frederick              2   \n",
       "3 2012-01-01 00:01:00   1282000006       Leonardtown              1   \n",
       "4 2012-01-01 01:01:00   1267000007             Essex              1   \n",
       "\n",
       "  DAY_OF_WEEK                       ROAD                INTERSECT_ROAD  \\\n",
       "0   SUNDAY      IS 00495 CAPITAL BELTWAY  IS 00270 EISENHOWER MEMORIAL   \n",
       "1   SUNDAY     MD 00090 OCEAN CITY EXPWY   CO 00220 ST MARTINS NECK RD   \n",
       "2   SUNDAY              MD 00765 MAIN ST              CO 00208 DUKE ST   \n",
       "3   SUNDAY      MD 00944 MERVELL DEAN RD       MD 00235 THREE NOTCH RD   \n",
       "4   SUNDAY        IS 00695 BALTO BELTWAY     IS 00083 HARRISBURG EXPWY   \n",
       "\n",
       "   DIST_FROM_INTERSECT DIST_DIRECTION       CITY_NAME  COUNTY_CODE  \\\n",
       "0                 0.00              U  Not Applicable         15.0   \n",
       "1                 0.25              W  Not Applicable         23.0   \n",
       "2               100.00              S  Not Applicable          4.0   \n",
       "3                10.00              E  Not Applicable         18.0   \n",
       "4               100.00              S  Not Applicable          3.0   \n",
       "\n",
       "  COUNTY_NAME  VEHICLE_COUNT PROP_DEST INJURY COLLISION_WITH_1  \\\n",
       "0  Montgomery            2.0       YES     NO              VEH   \n",
       "1   Worcester            1.0       YES     NO        FIXED OBJ   \n",
       "2     Calvert            1.0       YES     NO        FIXED OBJ   \n",
       "3   St. Marys            1.0       YES     NO        FIXED OBJ   \n",
       "4   Baltimore            2.0       YES     NO              VEH   \n",
       "\n",
       "  COLLISION_WITH_2  MONTH  \n",
       "0  OTHER-COLLISION      1  \n",
       "1  OTHER-COLLISION      1  \n",
       "2        FIXED OBJ      1  \n",
       "3  OTHER-COLLISION      1  \n",
       "4  OTHER-COLLISION      1  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google Maps API\n",
    "Now we will use the Google Maps API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from googlemaps.exceptions import TransportError\n",
    "import googlemaps\n",
    "from googlemaps.exceptions import HTTPError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "gmaps = googlemaps.Client(key=os.environ[\"YOUR GOOGLE API HERE\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function is used to get the official latitude and longitude for each county, by making calls to the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "county_names = list(set(data.COUNTY_NAME.unique()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Foursquare API\n",
    "\n",
    "Foursquare API documentation is [here](https://developer.foursquare.com/)\n",
    "\n",
    "1. Start a foursquare application and get your keys.\n",
    "2. For each crash, pull number of of bars (category \"Nightlife\") in 5km radius.\n",
    "3. Find a relationship between number of bars in the area and severity of the crash.\n",
    "4. (optional) Try to come up with other approaches to get more information out of the data. \n",
    "5. (optional) Think about the most generic way to approach the problem.\n",
    "\n",
    "Hints:\n",
    "\n",
    "* check out python package \"foursquare\"\n",
    "* what happens if the code fails?\n",
    "* what if you run out of requests? (check out [time](https://docs.python.org/2/library/time.html) package)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#set the keys\n",
    "foursquare_id = 'YOUR FOURSQUARE API HERE'\n",
    "foursquare_secret = \"YOUR FOURSQUARE SECRET CODE HERE\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#install and load the library\n",
    "from foursquare import Foursquare"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to set up the client in order to make calls to the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Foursquare(client_id = foursquare_id,\n",
    "                   client_secret = foursquare_secret)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will loop through the counties and obtain the number of bars within a 5km radius (up to a maximum of 50 bars). If the call quota is exceeded, then the code the operation will be paused for one hour to allow it to reset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Couldn't geocode param near: Not Applicable\n"
     ]
    }
   ],
   "source": [
    "number_of_bars = {}\n",
    "for county in county_names:\n",
    "    try:\n",
    "        response = client.venues.search({'near': county,\n",
    "                                         'limit': 50,\n",
    "                                         'intent': 'browse',\n",
    "                                        'radius': 5000,\n",
    "                                        'units': 'si',\n",
    "                                        'categoryId': '4d4b7105d754a06376d81259'})\n",
    "        number_of_bars[county] = len(response['venues'])\n",
    "    except Exception as e:\n",
    "        print (e)\n",
    "        if e == \"Quota exceeded\":\n",
    "            print (\"exceeded quota: waiting for an hour\")\n",
    "            time.sleep(3600)\n",
    "        number_of_bars[county] = -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Allegany': 30,\n",
       " 'Anne Arundel': 49,\n",
       " 'Baltimore': 50,\n",
       " 'Baltimore City': 50,\n",
       " 'Calvert': 0,\n",
       " 'Caroline': 6,\n",
       " 'Carroll': 22,\n",
       " 'Cecil': 28,\n",
       " 'Charles': 13,\n",
       " 'Dorchester': 23,\n",
       " 'Frederick': 35,\n",
       " 'Garrett': 12,\n",
       " 'Harford': 2,\n",
       " 'Howard': 9,\n",
       " 'Kent': 50,\n",
       " 'Montgomery': 50,\n",
       " 'Not Applicable': -1,\n",
       " 'Prince Georges': 29,\n",
       " 'Queen Annes': 8,\n",
       " 'Somerset': 41,\n",
       " 'St. Marys': 13,\n",
       " 'Talbot': 3,\n",
       " 'Unknown': 0,\n",
       " 'Washington': 50,\n",
       " 'Wicomico': 50,\n",
       " 'Worcester': 39}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "number_of_bars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame({'counties': list(number_of_bars.keys()), 'num_of_bars': list(number_of_bars.values())}).to_csv('number-of-bars.csv')\n",
    "              "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding a new column for the number of bars \n",
    "number_of_bars = pd.DataFrame({'county': list(number_of_bars.keys()), 'num_bars': list(number_of_bars.values())})\n",
    "data_df = pd.merge(data, number_of_bars, left_on='COUNTY_NAME', right_on='county', how='left')\n",
    "data_df.drop(columns=['county'], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to select a target variable. I will choose the 'INJURY' column, because this is a straightforward way to judge the severity of the crash."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting injuries to a binary mapping to judge severity\n",
    "data_df['severity'] = data_df['INJURY'].map({'YES':1, 'NO':0})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ACC_DATE_ACC_TIME</th>\n",
       "      <th>CASE_NUMBER</th>\n",
       "      <th>BARRACK</th>\n",
       "      <th>ACC_TIME_CODE</th>\n",
       "      <th>DAY_OF_WEEK</th>\n",
       "      <th>ROAD</th>\n",
       "      <th>INTERSECT_ROAD</th>\n",
       "      <th>DIST_FROM_INTERSECT</th>\n",
       "      <th>DIST_DIRECTION</th>\n",
       "      <th>CITY_NAME</th>\n",
       "      <th>COUNTY_CODE</th>\n",
       "      <th>COUNTY_NAME</th>\n",
       "      <th>VEHICLE_COUNT</th>\n",
       "      <th>PROP_DEST</th>\n",
       "      <th>INJURY</th>\n",
       "      <th>COLLISION_WITH_1</th>\n",
       "      <th>COLLISION_WITH_2</th>\n",
       "      <th>MONTH</th>\n",
       "      <th>num_bars</th>\n",
       "      <th>severity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2012-01-01 02:01:00</td>\n",
       "      <td>1363000002</td>\n",
       "      <td>Rockville</td>\n",
       "      <td>1</td>\n",
       "      <td>SUNDAY</td>\n",
       "      <td>IS 00495 CAPITAL BELTWAY</td>\n",
       "      <td>IS 00270 EISENHOWER MEMORIAL</td>\n",
       "      <td>0.00</td>\n",
       "      <td>U</td>\n",
       "      <td>Not Applicable</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Montgomery</td>\n",
       "      <td>2.0</td>\n",
       "      <td>YES</td>\n",
       "      <td>NO</td>\n",
       "      <td>VEH</td>\n",
       "      <td>OTHER-COLLISION</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2012-01-01 18:01:00</td>\n",
       "      <td>1296000023</td>\n",
       "      <td>Berlin</td>\n",
       "      <td>5</td>\n",
       "      <td>SUNDAY</td>\n",
       "      <td>MD 00090 OCEAN CITY EXPWY</td>\n",
       "      <td>CO 00220 ST MARTINS NECK RD</td>\n",
       "      <td>0.25</td>\n",
       "      <td>W</td>\n",
       "      <td>Not Applicable</td>\n",
       "      <td>23.0</td>\n",
       "      <td>Worcester</td>\n",
       "      <td>1.0</td>\n",
       "      <td>YES</td>\n",
       "      <td>NO</td>\n",
       "      <td>FIXED OBJ</td>\n",
       "      <td>OTHER-COLLISION</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2012-01-01 07:01:00</td>\n",
       "      <td>1283000016</td>\n",
       "      <td>Prince Frederick</td>\n",
       "      <td>2</td>\n",
       "      <td>SUNDAY</td>\n",
       "      <td>MD 00765 MAIN ST</td>\n",
       "      <td>CO 00208 DUKE ST</td>\n",
       "      <td>100.00</td>\n",
       "      <td>S</td>\n",
       "      <td>Not Applicable</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Calvert</td>\n",
       "      <td>1.0</td>\n",
       "      <td>YES</td>\n",
       "      <td>NO</td>\n",
       "      <td>FIXED OBJ</td>\n",
       "      <td>FIXED OBJ</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2012-01-01 00:01:00</td>\n",
       "      <td>1282000006</td>\n",
       "      <td>Leonardtown</td>\n",
       "      <td>1</td>\n",
       "      <td>SUNDAY</td>\n",
       "      <td>MD 00944 MERVELL DEAN RD</td>\n",
       "      <td>MD 00235 THREE NOTCH RD</td>\n",
       "      <td>10.00</td>\n",
       "      <td>E</td>\n",
       "      <td>Not Applicable</td>\n",
       "      <td>18.0</td>\n",
       "      <td>St. Marys</td>\n",
       "      <td>1.0</td>\n",
       "      <td>YES</td>\n",
       "      <td>NO</td>\n",
       "      <td>FIXED OBJ</td>\n",
       "      <td>OTHER-COLLISION</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2012-01-01 01:01:00</td>\n",
       "      <td>1267000007</td>\n",
       "      <td>Essex</td>\n",
       "      <td>1</td>\n",
       "      <td>SUNDAY</td>\n",
       "      <td>IS 00695 BALTO BELTWAY</td>\n",
       "      <td>IS 00083 HARRISBURG EXPWY</td>\n",
       "      <td>100.00</td>\n",
       "      <td>S</td>\n",
       "      <td>Not Applicable</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Baltimore</td>\n",
       "      <td>2.0</td>\n",
       "      <td>YES</td>\n",
       "      <td>NO</td>\n",
       "      <td>VEH</td>\n",
       "      <td>OTHER-COLLISION</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    ACC_DATE_ACC_TIME  CASE_NUMBER           BARRACK  ACC_TIME_CODE  \\\n",
       "0 2012-01-01 02:01:00   1363000002         Rockville              1   \n",
       "1 2012-01-01 18:01:00   1296000023            Berlin              5   \n",
       "2 2012-01-01 07:01:00   1283000016  Prince Frederick              2   \n",
       "3 2012-01-01 00:01:00   1282000006       Leonardtown              1   \n",
       "4 2012-01-01 01:01:00   1267000007             Essex              1   \n",
       "\n",
       "  DAY_OF_WEEK                       ROAD                INTERSECT_ROAD  \\\n",
       "0   SUNDAY      IS 00495 CAPITAL BELTWAY  IS 00270 EISENHOWER MEMORIAL   \n",
       "1   SUNDAY     MD 00090 OCEAN CITY EXPWY   CO 00220 ST MARTINS NECK RD   \n",
       "2   SUNDAY              MD 00765 MAIN ST              CO 00208 DUKE ST   \n",
       "3   SUNDAY      MD 00944 MERVELL DEAN RD       MD 00235 THREE NOTCH RD   \n",
       "4   SUNDAY        IS 00695 BALTO BELTWAY     IS 00083 HARRISBURG EXPWY   \n",
       "\n",
       "   DIST_FROM_INTERSECT DIST_DIRECTION       CITY_NAME  COUNTY_CODE  \\\n",
       "0                 0.00              U  Not Applicable         15.0   \n",
       "1                 0.25              W  Not Applicable         23.0   \n",
       "2               100.00              S  Not Applicable          4.0   \n",
       "3                10.00              E  Not Applicable         18.0   \n",
       "4               100.00              S  Not Applicable          3.0   \n",
       "\n",
       "  COUNTY_NAME  VEHICLE_COUNT PROP_DEST INJURY COLLISION_WITH_1  \\\n",
       "0  Montgomery            2.0       YES     NO              VEH   \n",
       "1   Worcester            1.0       YES     NO        FIXED OBJ   \n",
       "2     Calvert            1.0       YES     NO        FIXED OBJ   \n",
       "3   St. Marys            1.0       YES     NO        FIXED OBJ   \n",
       "4   Baltimore            2.0       YES     NO              VEH   \n",
       "\n",
       "  COLLISION_WITH_2  MONTH  num_bars  severity  \n",
       "0  OTHER-COLLISION      1        50         0  \n",
       "1  OTHER-COLLISION      1        39         0  \n",
       "2        FIXED OBJ      1         0         0  \n",
       "3  OTHER-COLLISION      1        13         0  \n",
       "4  OTHER-COLLISION      1        50         0  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will now create a new dataframe for my features and encode each feature into categoric variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Time</th>\n",
       "      <th>Day</th>\n",
       "      <th>Vehicles</th>\n",
       "      <th>One hit</th>\n",
       "      <th>Tws hits</th>\n",
       "      <th>Bars</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1363000002</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1296000023</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1283000016</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1282000006</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1267000007</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id  Time  Day  Vehicles  One hit  Tws hits  Bars\n",
       "0  1363000002     1    3       2.0        6         4    50\n",
       "1  1296000023     5    3       1.0        2         4    39\n",
       "2  1283000016     2    3       1.0        2         2     0\n",
       "3  1282000006     1    3       1.0        2         4    13\n",
       "4  1267000007     1    3       2.0        6         4    50"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "le = LabelEncoder()\n",
    "# crash severity\n",
    "feature_df = pd.DataFrame()\n",
    "feature_df['id'] = data_df['CASE_NUMBER']\n",
    "feature_df['Time'] = data_df['ACC_TIME_CODE']\n",
    "feature_df['Day'] = le.fit_transform(data_df['DAY_OF_WEEK'])\n",
    "feature_df['Vehicles'] = data_df['VEHICLE_COUNT'].fillna(0)\n",
    "feature_df['One hit'] = le.fit_transform(data_df['COLLISION_WITH_1'])\n",
    "feature_df['Tws hits'] = le.fit_transform(data_df['COLLISION_WITH_2'])\n",
    "feature_df['Bars'] = data_df['num_bars']\n",
    "\n",
    "feature_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id          0\n",
       "Time        0\n",
       "Day         0\n",
       "Vehicles    0\n",
       "One hit     0\n",
       "Tws hits    0\n",
       "Bars        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's make sure that there are no missing values because they will cause the algorithm to crash\n",
    "feature_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will now use the Scikit-learn random forest classifier to fit a model and use that model to determine the importance of each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- COMPLETE ---\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "# Sets up a classifier and fits a model to all features of the dataset\n",
    "clf = RandomForestClassifier(n_estimators=150, max_depth=8, min_samples_leaf=4, max_features=0.2, n_jobs=-1, random_state=0)\n",
    "clf.fit(feature_df.drop(['id'],axis=1), data_df['severity'])\n",
    "# We need a list of features as well\n",
    "features = feature_df.drop(['id'],axis=1).columns.values\n",
    "print(\"--- COMPLETE ---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the following code from Anisotropic's kernal (https://www.kaggle.com/arthurtok/interactive-porto-insights-a-plot-ly-tutorial), we can use Plotly to create a nice horizontal bar chart for visualising the ranking of the most important features for determing the severity of crash."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "marker": {
          "cauto": true,
          "cmax": 0.2963757717862322,
          "cmin": 0.08076882138658228,
          "color": [
           0.08076882138658228,
           0.1059485976976364,
           0.13099807181047302,
           0.15069375016064962,
           0.23521498715842667,
           0.2963757717862322
          ],
          "colorscale": "Viridis",
          "reversescale": true
         },
         "name": "Random Forest Feature importance",
         "orientation": "h",
         "type": "bar",
         "uid": "db9b2e",
         "x": [
          0.08076882138658228,
          0.1059485976976364,
          0.13099807181047302,
          0.15069375016064962,
          0.23521498715842667,
          0.2963757717862322
         ],
         "y": [
          "Day",
          "Time",
          "Bars",
          "Vehicles",
          "Tws hits",
          "One hit"
         ]
        }
       ],
       "layout": {
        "height": 1500,
        "title": "Ranking of most influential features",
        "width": 900,
        "xaxis": {
         "autorange": true,
         "range": [
          0,
          0.31197449661708654
         ],
         "type": "linear"
        },
        "yaxis": {
         "autorange": true,
         "range": [
          -0.5,
          5.5
         ],
         "showgrid": false,
         "showline": false,
         "showticklabels": true,
         "type": "category"
        }
       }
      },
      "text/html": [
       "<div id=\"98870fdf-7bc5-4f81-acfd-907b83f53c18\" style=\"height: 1500px; width: 900px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"98870fdf-7bc5-4f81-acfd-907b83f53c18\", [{\"type\": \"bar\", \"x\": [0.08076882138658228, 0.1059485976976364, 0.13099807181047302, 0.15069375016064962, 0.23521498715842667, 0.2963757717862322], \"y\": [\"Day\", \"Time\", \"Bars\", \"Vehicles\", \"Tws hits\", \"One hit\"], \"marker\": {\"color\": [0.08076882138658228, 0.1059485976976364, 0.13099807181047302, 0.15069375016064962, 0.23521498715842667, 0.2963757717862322], \"colorscale\": \"Viridis\", \"reversescale\": true}, \"name\": \"Random Forest Feature importance\", \"orientation\": \"h\"}], {\"title\": \"Ranking of most influential features\", \"width\": 900, \"height\": 1500, \"yaxis\": {\"showgrid\": false, \"showline\": false, \"showticklabels\": true}}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"98870fdf-7bc5-4f81-acfd-907b83f53c18\" style=\"height: 1500px; width: 900px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"98870fdf-7bc5-4f81-acfd-907b83f53c18\", [{\"type\": \"bar\", \"x\": [0.08076882138658228, 0.1059485976976364, 0.13099807181047302, 0.15069375016064962, 0.23521498715842667, 0.2963757717862322], \"y\": [\"Day\", \"Time\", \"Bars\", \"Vehicles\", \"Tws hits\", \"One hit\"], \"marker\": {\"color\": [0.08076882138658228, 0.1059485976976364, 0.13099807181047302, 0.15069375016064962, 0.23521498715842667, 0.2963757717862322], \"colorscale\": \"Viridis\", \"reversescale\": true}, \"name\": \"Random Forest Feature importance\", \"orientation\": \"h\"}], {\"title\": \"Ranking of most influential features\", \"width\": 900, \"height\": 1500, \"yaxis\": {\"showgrid\": false, \"showline\": false, \"showticklabels\": true}}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import plotly.offline as py\n",
    "py.init_notebook_mode(connected=True)\n",
    "import plotly.graph_objs as go\n",
    "import plotly.tools as tls\n",
    "\n",
    "x, y = (list(x) for x in zip(*sorted(zip(clf.feature_importances_, features), \n",
    "                                                            reverse = False)))\n",
    "trace2 = go.Bar(\n",
    "    x=x ,\n",
    "    y=y,\n",
    "    marker=dict(\n",
    "        color=x,\n",
    "        colorscale = 'Viridis',\n",
    "        reversescale = True\n",
    "    ),\n",
    "    name='Random Forest Feature importance',\n",
    "    orientation='h',\n",
    ")\n",
    "\n",
    "layout = dict(\n",
    "    title='Ranking of most influential features',\n",
    "     width = 900, height = 1500,\n",
    "    yaxis=dict(\n",
    "        showgrid=False,\n",
    "        showline=False,\n",
    "        showticklabels=True,\n",
    "    ))\n",
    "\n",
    "fig1 = go.Figure(data=[trace2])\n",
    "fig1['layout'].update(layout)\n",
    "py.iplot(fig1, filename='plots')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DarkSky API\n",
    "\n",
    "DarkSky API documentation is [here](https://darksky.net/dev/docs/time-machine)\n",
    "\n",
    "1. Sign up for FREE api key.\n",
    "2. For each crush, get the weather for the location and time.\n",
    "3. Find a relationship between the weather and severity of the crash.\n",
    "\n",
    "Hints:\n",
    "\n",
    "* There is an API limit (perhaps 1000 calls)\n",
    "* use \"Time Machine\" request in DarkSky API\n",
    "* for sending HTTP requests check out \"requests\" library [here](http://docs.python-requests.org/en/master/)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The time needs to be converted to unix in order for the DarkSky API to recognise it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "def time_to_unix(t):\n",
    "    return time.mktime(t.timetuple())\n",
    "\n",
    "# changing to unix time\n",
    "data_sample_df = data_df.sample(n=1000, random_state=0)\n",
    "data_sample_df['UNIX_TIME'] = data_sample_df['ACC_DATE_ACC_TIME'].apply(time_to_unix).astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now need to get the latitude and longitude for each county in order for the API to provide local weather for each crash."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_lat_lng(data, state, place_col):\n",
    "    places = list(set(data[place_col].unique()))\n",
    "    \n",
    "    geo_dict = {'place':[], 'lat':[], 'lng':[]}\n",
    "    for i in range(len(places)):\n",
    "        place = places[i] + ', ' + state\n",
    "        try:    \n",
    "            lat = gmaps.geocode(place)[0]['geometry']['location']['lat']\n",
    "            lng = gmaps.geocode(place)[0]['geometry']['location']['lng']\n",
    "            geo_dict['place'].append(places[i])\n",
    "            geo_dict['lat'].append(lat)\n",
    "            geo_dict['lng'].append(lng)\n",
    "        except:\n",
    "            geo_dict['place'].append(None)\n",
    "            geo_dict['lat'].append(None)\n",
    "            geo_dict['lng'].append(None)\n",
    "\n",
    "    geo_df = pd.DataFrame(geo_dict)\n",
    "    return pd.merge(data, geo_df, left_on='COUNTY_NAME', right_on='place', how='left')\n",
    "\n",
    "data_sample_geo_df = get_lat_lng(data_sample_df, 'Maryland', 'COUNTY_NAME')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prince Georges    180\n",
       "Harford            94\n",
       "Baltimore          91\n",
       "Frederick          81\n",
       "Montgomery         66\n",
       "Anne Arundel       65\n",
       "Carroll            49\n",
       "Charles            45\n",
       "Cecil              43\n",
       "Washington         38\n",
       "Howard             28\n",
       "Allegany           27\n",
       "Queen Annes        26\n",
       "Wicomico           26\n",
       "Caroline           25\n",
       "Somerset           18\n",
       "Talbot             17\n",
       "Dorchester         17\n",
       "Calvert            16\n",
       "Worcester          16\n",
       "Garrett            12\n",
       "St. Marys          10\n",
       "Kent               10\n",
       "Name: COUNTY_NAME, dtype: int64"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_sample_geo_df.COUNTY_NAME.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Once you have signed up on DarkSky.net, you will be given an API key, which you need to insert here.\n",
    "api_key = \"YOUR DARKSKY API HERE\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to make requests to the API for each of the crash instances in our sample. I am simply appending each entry into a dictionary with the place, coordinates, time and returned results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import time\n",
    "\n",
    "weather_data = {'place': [], 'lat': [], 'lng': [], 'time': [], 'result': []}\n",
    "\n",
    "for crash in data_sample_geo_df.iterrows():\n",
    "    place = crash[1]['COUNTY_NAME']\n",
    "    lat = crash[1]['lat']\n",
    "    lng = crash[1]['lng']\n",
    "    t = crash[1]['UNIX_TIME']\n",
    "    # https://api.darksky.net/forecast/[key]/[latitude],[longitude],[time]\n",
    "    request = 'https://api.darksky.net/forecast/' + api_key + '/' + str(lat) + ',' + str(lng) + ',' + str(t)\n",
    "    \n",
    "    try:\n",
    "        result = requests.get(request).content\n",
    "        \n",
    "        weather_data['place'].append(place)\n",
    "        weather_data['lat'].append(lat)\n",
    "        weather_data['lng'].append(lng)\n",
    "        weather_data['time'].append(t)\n",
    "        weather_data['result'].append(result)\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "        weather_data['place'].append('')\n",
    "        weather_data['lat'].append('')\n",
    "        weather_data['lng'].append('')\n",
    "        weather_data['time'].append('')\n",
    "        weather_data['result'].append('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df = pd.concat([data_sample_geo_df[['CASE_NUMBER']], pd.DataFrame(weather_data).reset_index(drop=True)], axis=1)\n",
    "# I like to save results like these in a CSV file, because there is a limit on the number of API calls\n",
    "weather_df.to_csv('weather-data.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After going through one of the JSON files that was returned to me, I found the section that I want (\"currently\"). It contains the overall weather conditions, precipitation type (if any) and most importantly, the chance of rain. If the chance is greater than 50%, then I am satisfied that if was raining at the time for the sake of this exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expecting value: line 1 column 1 (char 0)\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "def extract_data(x):\n",
    "    try:\n",
    "        d = json.loads(x)\n",
    "        res = d['currently']['precipProbability']\n",
    "    except Exception as e:\n",
    "        print (e)\n",
    "        res = ''\n",
    "    return res\n",
    "\n",
    "weather_df['precipProb'] = weather_df['result'].apply(extract_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One line failed, so I will insert a zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_df['precipProb'] = weather_df['precipProb'].replace('', 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_final = data_sample_df.merge(weather_df, left_on='CASE_NUMBER', right_on='CASE_NUMBER', how='outer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PointbiserialrResult(correlation=-0.01817849772457154, pvalue=0.56584394823957873)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import pointbiserialr\n",
    "\n",
    "corr = pointbiserialr(df_final['severity'], df_final['precipProb'])\n",
    "corr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding weather to feature set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The precipitation probability feature can be included in a new feature table, along with the other features we used before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Time</th>\n",
       "      <th>Day</th>\n",
       "      <th>Vehicles</th>\n",
       "      <th>One hit</th>\n",
       "      <th>Tws hits</th>\n",
       "      <th>Bars</th>\n",
       "      <th>rain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1251037421</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>35</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1251029176</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>35</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1280008862</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1294006103</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1283001115</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           id  Time  Day  Vehicles  One hit  Tws hits  Bars  rain\n",
       "0  1251037421     3    6       2.0        6         2    35   0.0\n",
       "1  1251029176     6    1       2.0        6         2    35   0.0\n",
       "2  1280008862     5    0       2.0        6         4     8   0.0\n",
       "3  1294006103     2    4       1.0        4         0    12   0.0\n",
       "4  1283001115     6    0       1.0        2         1     0   0.0"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "le = LabelEncoder()\n",
    "# crash severity\n",
    "feature_df = pd.DataFrame()\n",
    "feature_df['id'] = df_final['CASE_NUMBER']\n",
    "feature_df['Time'] = df_final['ACC_TIME_CODE']\n",
    "feature_df['Day'] = le.fit_transform(df_final['DAY_OF_WEEK'])\n",
    "feature_df['Vehicles'] = df_final['VEHICLE_COUNT'].fillna(0)\n",
    "feature_df['One hit'] = le.fit_transform(df_final['COLLISION_WITH_1'])\n",
    "feature_df['Tws hits'] = le.fit_transform(df_final['COLLISION_WITH_2'])\n",
    "feature_df['Bars'] = df_final['num_bars']\n",
    "feature_df['rain'] = df_final['precipProb']\n",
    "\n",
    "feature_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id          0\n",
       "Time        0\n",
       "Day         0\n",
       "Vehicles    0\n",
       "One hit     0\n",
       "Tws hits    0\n",
       "Bars        0\n",
       "rain        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's make sure that there are no missing values because they will cause the algorithm to crash\n",
    "feature_df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I would like to use the Scikit-learn random forest classifier again to fit a model and use that model to determine the importance of each feature, now including the precipitation probability feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- COMPLETE ---\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "# Sets up a classifier and fits a model to all features of the dataset\n",
    "clf = RandomForestClassifier(n_estimators=150, max_depth=8, min_samples_leaf=4, max_features=0.2, n_jobs=-1, random_state=0)\n",
    "clf.fit(feature_df.drop(['id'],axis=1), data_sample_df['severity'])\n",
    "# We need a list of features as well\n",
    "features = feature_df.drop(['id'],axis=1).columns.values\n",
    "print(\"--- COMPLETE ---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, I will plot a bar chart to compare features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "marker": {
          "cauto": true,
          "cmax": 0.18924724467542287,
          "cmin": 0.0855069175420826,
          "color": [
           0.0855069175420826,
           0.11677971139123805,
           0.1279283892945433,
           0.1408541979694031,
           0.1556637796381357,
           0.18401975948917426,
           0.18924724467542287
          ],
          "colorscale": "Viridis",
          "reversescale": true
         },
         "name": "Random Forest Feature importance",
         "orientation": "h",
         "type": "bar",
         "uid": "ace661",
         "x": [
          0.0855069175420826,
          0.11677971139123805,
          0.1279283892945433,
          0.1408541979694031,
          0.1556637796381357,
          0.18401975948917426,
          0.18924724467542287
         ],
         "y": [
          "Vehicles",
          "Day",
          "rain",
          "Time",
          "One hit",
          "Bars",
          "Tws hits"
         ]
        }
       ],
       "layout": {
        "height": 1500,
        "title": "Ranking of most influential features",
        "width": 900,
        "xaxis": {
         "autorange": true,
         "range": [
          0,
          0.19920762597412933
         ],
         "type": "linear"
        },
        "yaxis": {
         "autorange": true,
         "range": [
          -0.5,
          6.5
         ],
         "showgrid": false,
         "showline": false,
         "showticklabels": true,
         "type": "category"
        }
       }
      },
      "text/html": [
       "<div id=\"e114f9bc-a5a2-4eea-8bc9-8a86d57db347\" style=\"height: 1500px; width: 900px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"e114f9bc-a5a2-4eea-8bc9-8a86d57db347\", [{\"type\": \"bar\", \"x\": [0.0855069175420826, 0.11677971139123805, 0.1279283892945433, 0.1408541979694031, 0.1556637796381357, 0.18401975948917426, 0.18924724467542287], \"y\": [\"Vehicles\", \"Day\", \"rain\", \"Time\", \"One hit\", \"Bars\", \"Tws hits\"], \"marker\": {\"color\": [0.0855069175420826, 0.11677971139123805, 0.1279283892945433, 0.1408541979694031, 0.1556637796381357, 0.18401975948917426, 0.18924724467542287], \"colorscale\": \"Viridis\", \"reversescale\": true}, \"name\": \"Random Forest Feature importance\", \"orientation\": \"h\"}], {\"title\": \"Ranking of most influential features\", \"width\": 900, \"height\": 1500, \"yaxis\": {\"showgrid\": false, \"showline\": false, \"showticklabels\": true}}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"e114f9bc-a5a2-4eea-8bc9-8a86d57db347\" style=\"height: 1500px; width: 900px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"e114f9bc-a5a2-4eea-8bc9-8a86d57db347\", [{\"type\": \"bar\", \"x\": [0.0855069175420826, 0.11677971139123805, 0.1279283892945433, 0.1408541979694031, 0.1556637796381357, 0.18401975948917426, 0.18924724467542287], \"y\": [\"Vehicles\", \"Day\", \"rain\", \"Time\", \"One hit\", \"Bars\", \"Tws hits\"], \"marker\": {\"color\": [0.0855069175420826, 0.11677971139123805, 0.1279283892945433, 0.1408541979694031, 0.1556637796381357, 0.18401975948917426, 0.18924724467542287], \"colorscale\": \"Viridis\", \"reversescale\": true}, \"name\": \"Random Forest Feature importance\", \"orientation\": \"h\"}], {\"title\": \"Ranking of most influential features\", \"width\": 900, \"height\": 1500, \"yaxis\": {\"showgrid\": false, \"showline\": false, \"showticklabels\": true}}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import plotly.offline as py\n",
    "py.init_notebook_mode(connected=True)\n",
    "import plotly.graph_objs as go\n",
    "import plotly.tools as tls\n",
    "\n",
    "x, y = (list(x) for x in zip(*sorted(zip(clf.feature_importances_, features), \n",
    "                                                            reverse = False)))\n",
    "trace2 = go.Bar(\n",
    "    x=x ,\n",
    "    y=y,\n",
    "    marker=dict(\n",
    "        color=x,\n",
    "        colorscale = 'Viridis',\n",
    "        reversescale = True\n",
    "    ),\n",
    "    name='Random Forest Feature importance',\n",
    "    orientation='h',\n",
    ")\n",
    "\n",
    "layout = dict(\n",
    "    title='Ranking of most influential features',\n",
    "     width = 900, height = 1500,\n",
    "    yaxis=dict(\n",
    "        showgrid=False,\n",
    "        showline=False,\n",
    "        showticklabels=True,\n",
    "    ))\n",
    "\n",
    "fig1 = go.Figure(data=[trace2])\n",
    "fig1['layout'].update(layout)\n",
    "py.iplot(fig1, filename='plots')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the ranking order is quite different to the last one. The difference here is that a sample of only 1,000 accidents was used instead of the 18,000 accidents that were in the initial dataset. Even though I took a randomised sample of instances, there is a big discrepancies in the data. According to this sample dataset, the number of bars in the area is the biggest determining factor in an accident, quite a bit more than the weather. It would be nice to have weather data for all samples in the dataset, but this requires upgrading the DarkSpy membership to get more calls or pausing the API calls for an hour at time and waiting for more than 18 hours to get a result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Google Maps plot of the accidents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will now obtain the latitude and longitudes of the nearest intersections of 100 accidents and plot\n",
    "them on a Google map, using the Google Map Plotter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [],
   "source": [
    "crash_plot_df = data_sample_geo_df.sample(n=100, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [],
   "source": [
    "crash_plot_df['intersection'] = crash_plot_df['ROAD'] + ' ' + crash_plot_df['INTERSECT_ROAD'] + ' ' + crash_plot_df['COUNTY_NAME'] + ' ' + 'MARYLAND'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TMakes calls to the gmaps API to get latitude information for an address\n",
    "def get_lat(x):\n",
    "    try: \n",
    "        return gmaps.geocode(x)[0]['geometry']['location']['lat'] \n",
    "    except:\n",
    "        return ''\n",
    "    \n",
    "# Makes calls to the gmaps API to get longitute information for an address\n",
    "def get_lng(x):\n",
    "    try: \n",
    "        return gmaps.geocode(x)[0]['geometry']['location']['lng'] \n",
    "    except:\n",
    "        return ''\n",
    "\n",
    "# Lat and lon info is then added to the dataframe\n",
    "crash_plot_df['crash_lat'] = crash_plot_df['intersection'].apply(get_lat)\n",
    "crash_plot_df['crash_lng'] = crash_plot_df['intersection'].apply(get_lng)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, I will plot all of the points and use the gmplot API to create an HTML file with a map of Maryland and the crash sites."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gmplot import gmplot\n",
    "\n",
    "# Coordinates for Maryland\n",
    "state_lat = gmaps.geocode('Maryland')[0]['geometry']['location']['lat']\n",
    "state_lng = gmaps.geocode('Maryland')[0]['geometry']['location']['lng']\n",
    "\n",
    "# Place map\n",
    "gmap = gmplot.GoogleMapPlotter(state_lat, state_lng, 8)\n",
    "\n",
    "# Coordinates for Counties\n",
    "county_lats = crash_plot_df['lat'].values.tolist()\n",
    "county_lngs = crash_plot_df['lng'].values.tolist()\n",
    "\n",
    "# Coordinates for crashes\n",
    "crash_lats = crash_plot_df['crash_lat'].values.tolist()\n",
    "crash_lngs = crash_plot_df['crash_lng'].values.tolist()\n",
    "\n",
    "# Scatter points\n",
    "gmap.scatter(crash_lats, crash_lngs, 'red', size=2500, marker=True)\n",
    "\n",
    "# Draw\n",
    "gmap.draw(\"my_map.html\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}