final-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "71fd8080c77935a0e21d4944691108b4280ed8f6",
    "colab_type": "text",
    "id": "5j5CvPN43ctS"
   },
   "source": [
    "# Road Accident Prediction and Classification\n",
    "###### Abdul Wahed and Abrar      \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sFBK3MEq3ctU"
   },
   "outputs": [],
   "source": [
    "#importing OS module which for directory acccess and view\n",
    "import os\n",
    "\n",
    "# print(os.getcwd())\n",
    "# os.getcwd()\n",
    "# print(os.listdir('..'))\n",
    "# print(os.listdir('../anacond'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "4f3c71060eb916343706bb2d8a500ba979e331dc",
    "colab_type": "text",
    "id": "L34I0rla3cta"
   },
   "source": [
    "### Introduction\n",
    "\n",
    " There are some questions that can be answered using this data such as -\n",
    "- What are the regions or areas with most frequent accidents?\n",
    "- What kind of street or highways are more liekly to have accidents?\n",
    "- What are the age group are most likely to be involved in accidents?\n",
    "- What are the areas with higher accident severity or lower accident severity?\n",
    "   \n",
    "There are endless questions that can be answered with this dataset. We will be answering few of the questions as I mentioned above. We will also figure out some way to implement the machine learning on this dataset and see what we can come up with.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "28c6e782278d576238ac2fd5fbddfc88adc61796",
    "colab_type": "text",
    "id": "wjlef1AJ3ctb"
   },
   "source": [
    "## Importing Data and cleaning\n",
    "- We import three files to perform analysis on this data. This data is consist of three files that are accidents, casualities and vehicles. However, we have one more file which is general information about the traffic count for year 2000 to 2015. We can use general traffic information data for machine learning part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_uuid": "43f2cbb55a0f5e95b02c7ee99d9e416f83836dfd",
    "colab": {},
    "colab_type": "code",
    "id": "CxF1GeSx3ctc"
   },
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "# from mpl_toolkits.basemap import Basemap\n",
    "from sklearn.model_selection import TimeSeriesSplit\n",
    "plt.style.use('ggplot')\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "0b93ca1a20c07af055065343e30fb64c37dcadd1",
    "colab": {},
    "colab_type": "code",
    "id": "twJwt-op3ctg"
   },
   "outputs": [],
   "source": [
    "## ---------METHOD 1 (FROM GITHUB)---------------------\n",
    "#!curl https://raw.githubusercontent.com/abdulwahed786/final-yr-projectqA/master/Accidents.csv\n",
    "# dataframe = pd.read_csv(!curl https://raw.githubusercontent.com/abdulwahed786/final-yr-projectqA/master/Accidents.csv)\n",
    "# dataframe.head()\n",
    "# import requests\n",
    "\n",
    "# url=\"https://raw.githubusercontent.com/abdulwahed786/final-yr-projectqA/master/Accidents.csv\"\n",
    "# s=requests.get(url).content\n",
    "# c=pd.read_csv(s)\n",
    "#accidents = pd.read_csv('https://raw.githubusercontent.com/abdulwahed786/final-yr-projectqA/master/Accidents.csv',index_col='Accident_Index')\n",
    "#accidents = pd.read_csv('Accidents.csv',index_col='Accident_Index')\n",
    "#casualties= pd.read_csv('https://raw.githubusercontent.com/abdulwahed786/final-yr-projectqA/master/Casualties.csv' , error_bad_lines=False,index_col='Accident_Index',warn_bad_lines=False)\n",
    "#vehicles= pd.read_csv('https://raw.githubusercontent.com/abdulwahed786/final-yr-projectqA/master/Vehicles.csv', error_bad_lines=False,index_col='Accident_Index',warn_bad_lines=False)\n",
    "#general_info = pd.read_csv('ukTrafficAADF.csv')\n",
    "\n",
    "\n",
    "## ---------METHOD 2 (FROM AZURE)---------------------\n",
    "# #importing data from azure workspace\n",
    "# from azureml import Workspace\n",
    "# ws = Workspace(\n",
    "#     workspace_id='f8311c4e9dd942c4b5fb2b322c164a59',\n",
    "#     authorization_token='tk6XlAPlYCw+cAbzvQsMREYwgR6OHrY4o/1Xjg82Rqlt+aHo89SXHtLseUc0Dn3VYrQzl+3q8UTzIgnw5b36EA==',\n",
    "#     endpoint='https://studioapi.azureml.net'\n",
    "# )\n",
    "# ds = ws.datasets['Vehicles0515.csv']\n",
    "# frame = ds.to_dataframe()\n",
    "\n",
    "# from azureml import Workspace\n",
    "# ws = Workspace(\n",
    "#     workspace_id='f8311c4e9dd942c4b5fb2b322c164a59',\n",
    "#     authorization_token='tk6XlAPlYCw+cAbzvQsMREYwgR6OHrY4o/1Xjg82Rqlt+aHo89SXHtLseUc0Dn3VYrQzl+3q8UTzIgnw5b36EA==',\n",
    "#     endpoint='https://studioapi.azureml.net'\n",
    "# )\n",
    "# ds = ws.datasets['Accidents0515.csv']\n",
    "# accidents = ds.to_dataframe().set_index('Accident_Index')\n",
    "\n",
    "# dsc = ws.datasets['Casualties0515.csv']\n",
    "# casualties = dsc.to_dataframe()\n",
    "\n",
    "#accidents.set_index('Accident_Index')\n",
    "#accidents = pd.read_csv(frame,index_col='Accident_Index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "XoFn6i1D3ctj",
    "outputId": "c0e72ba2-9275-4ad3-a880-2d46afaa33ac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tqdm\n",
      "  Downloading https://files.pythonhosted.org/packages/6c/4b/c38b5144cf167c4f52288517436ccafefe9dc01b8d1c190e18a6b154cd4a/tqdm-4.31.1-py2.py3-none-any.whl (48kB)\n",
      "Installing collected packages: tqdm\n",
      "Successfully installed tqdm-4.31.1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 18.0, however version 19.0.3 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "# using python package TQDM to download dataset locally on colab \n",
    "!pip install tqdm\n",
    "import requests\n",
    "import os\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EmOuXLcU3ctm"
   },
   "outputs": [],
   "source": [
    "# function for input to tqdm\n",
    "def download_dataset(file_url, name):\n",
    "    r = requests.get(file_url, stream=True) \n",
    "\n",
    "    with open(name, \"wb\") as file: \n",
    "        for chunk in tqdm(r.iter_content(chunk_size=1024)): \n",
    "             if chunk: file.write(chunk)\n",
    "                \n",
    "    print('Download complete.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "id": "JnLQMMnF3ctp",
    "outputId": "648794fa-ccd4-4b72-bdd8-6cf10361e4ec"
   },
   "outputs": [],
   "source": [
    "# download_dataset(\"https://bitbucket.org/abdulwahed11314/accidents-data/raw/b7add9860d310171bca48bcaefeae37fe5157ac3/CasualtiesBig.csv\", 'casualties.csv')\n",
    "# download_dataset(\"https://bitbucket.org/abdulwahed11314/accidents-data/raw/b7add9860d310171bca48bcaefeae37fe5157ac3/AccidentsBig.csv\", 'accidents.csv')\n",
    "# download_dataset(\"https://bitbucket.org/abdulwahed11314/accidents-data/raw/b7add9860d310171bca48bcaefeae37fe5157ac3/VehiclesBig.csv\", 'vehicles.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "AvuWU1ct3ctt",
    "outputId": "0b66ccb2-405f-4e84-bbb9-4810f721da00"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['.ipynb_checkpoints', 'accidents-data', 'AccidentsBig.csv', 'azure-FULL-uk-traffic-accidents.ipynb', 'azuremlFULL-uk-traffic-accidents.ipynb', 'casualties.csv', 'CasualtiesBig.csv', 'colabazuremlFULL_uk_traffic_accidents.ipynb', 'final-1.ipynb', 'full-import.ipynb', 'FULL-uk-traffic-accidents.ipynb', 'i-data', 'litemodel.sav', 'others report', 'outputs', 'RAPC-outputs.pptx', 'review-2-traffic-accidents.ipynb', 'review-3-traffic-accidents.ipynb', 'SampleIrisClassifierNotebook_py.ipynb', 'saved-colabazuremlFULL_uk_traffic_accidents.ipynb', 'savedmodel.sav', 'Untitled.ipynb', 'VehiclesBig.csv']\n"
     ]
    }
   ],
   "source": [
    "print(os.listdir('.'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "qN7Vcg6j3ctx",
    "outputId": "eeded7a6-c7e0-4a82-98ef-bd66d7c13bc6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded\n"
     ]
    }
   ],
   "source": [
    "accidents = pd.read_csv('accidentsBig.csv',index_col='Accident_Index')\n",
    "vehicles= pd.read_csv('vehiclesBig.csv', error_bad_lines=False,index_col='Accident_Index',warn_bad_lines=False)\n",
    "casualties = pd.read_csv('casualtiesBig.csv', error_bad_lines=False, index_col='Accident_Index', warn_bad_lines=False)\n",
    "print('Loaded')\n",
    "\n",
    "# accidents=accidents.head(200000)\n",
    "\n",
    "# vehicles=vehicles.head(200000)\n",
    "\n",
    "# casualties=casualties.head(200000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_uuid": "d7ceba9967f9674832cb2d88396be5477b718ec4",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 352
    },
    "colab_type": "code",
    "id": "W76Qu8fG3ct1",
    "outputId": "96101f9d-2f55-485d-ee60-8ea732469802"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accidents\n",
      "size= 55200243\n",
      "(1780653, 31)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Location_Easting_OSGR</th>\n",
       "      <th>Location_Northing_OSGR</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Police_Force</th>\n",
       "      <th>Accident_Severity</th>\n",
       "      <th>Number_of_Vehicles</th>\n",
       "      <th>Number_of_Casualties</th>\n",
       "      <th>Date</th>\n",
       "      <th>Day_of_Week</th>\n",
       "      <th>...</th>\n",
       "      <th>Pedestrian_Crossing-Human_Control</th>\n",
       "      <th>Pedestrian_Crossing-Physical_Facilities</th>\n",
       "      <th>Light_Conditions</th>\n",
       "      <th>Weather_Conditions</th>\n",
       "      <th>Road_Surface_Conditions</th>\n",
       "      <th>Special_Conditions_at_Site</th>\n",
       "      <th>Carriageway_Hazards</th>\n",
       "      <th>Urban_or_Rural_Area</th>\n",
       "      <th>Did_Police_Officer_Attend_Scene_of_Accident</th>\n",
       "      <th>LSOA_of_Accident_Location</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Accident_Index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200501BS00001</th>\n",
       "      <td>525680.0</td>\n",
       "      <td>178240.0</td>\n",
       "      <td>-0.191170</td>\n",
       "      <td>51.489096</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>04/01/2005</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>E01002849</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00002</th>\n",
       "      <td>524170.0</td>\n",
       "      <td>181650.0</td>\n",
       "      <td>-0.211708</td>\n",
       "      <td>51.520075</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>05/01/2005</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>E01002909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00003</th>\n",
       "      <td>524520.0</td>\n",
       "      <td>182240.0</td>\n",
       "      <td>-0.206458</td>\n",
       "      <td>51.525301</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>06/01/2005</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>E01002857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00004</th>\n",
       "      <td>526900.0</td>\n",
       "      <td>177530.0</td>\n",
       "      <td>-0.173862</td>\n",
       "      <td>51.482442</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>07/01/2005</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>E01002840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00005</th>\n",
       "      <td>528060.0</td>\n",
       "      <td>179040.0</td>\n",
       "      <td>-0.156618</td>\n",
       "      <td>51.495752</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>10/01/2005</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>E01002863</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Location_Easting_OSGR  Location_Northing_OSGR  Longitude  \\\n",
       "Accident_Index                                                             \n",
       "200501BS00001                525680.0                178240.0  -0.191170   \n",
       "200501BS00002                524170.0                181650.0  -0.211708   \n",
       "200501BS00003                524520.0                182240.0  -0.206458   \n",
       "200501BS00004                526900.0                177530.0  -0.173862   \n",
       "200501BS00005                528060.0                179040.0  -0.156618   \n",
       "\n",
       "                 Latitude  Police_Force  Accident_Severity  \\\n",
       "Accident_Index                                               \n",
       "200501BS00001   51.489096             1                  2   \n",
       "200501BS00002   51.520075             1                  3   \n",
       "200501BS00003   51.525301             1                  3   \n",
       "200501BS00004   51.482442             1                  3   \n",
       "200501BS00005   51.495752             1                  3   \n",
       "\n",
       "                Number_of_Vehicles  Number_of_Casualties        Date  \\\n",
       "Accident_Index                                                         \n",
       "200501BS00001                    1                     1  04/01/2005   \n",
       "200501BS00002                    1                     1  05/01/2005   \n",
       "200501BS00003                    2                     1  06/01/2005   \n",
       "200501BS00004                    1                     1  07/01/2005   \n",
       "200501BS00005                    1                     1  10/01/2005   \n",
       "\n",
       "                Day_of_Week            ...              \\\n",
       "Accident_Index                         ...               \n",
       "200501BS00001             3            ...               \n",
       "200501BS00002             4            ...               \n",
       "200501BS00003             5            ...               \n",
       "200501BS00004             6            ...               \n",
       "200501BS00005             2            ...               \n",
       "\n",
       "               Pedestrian_Crossing-Human_Control  \\\n",
       "Accident_Index                                     \n",
       "200501BS00001                                  0   \n",
       "200501BS00002                                  0   \n",
       "200501BS00003                                  0   \n",
       "200501BS00004                                  0   \n",
       "200501BS00005                                  0   \n",
       "\n",
       "                Pedestrian_Crossing-Physical_Facilities Light_Conditions  \\\n",
       "Accident_Index                                                             \n",
       "200501BS00001                                         1                1   \n",
       "200501BS00002                                         5                4   \n",
       "200501BS00003                                         0                4   \n",
       "200501BS00004                                         0                1   \n",
       "200501BS00005                                         0                7   \n",
       "\n",
       "                Weather_Conditions  Road_Surface_Conditions  \\\n",
       "Accident_Index                                                \n",
       "200501BS00001                    2                        2   \n",
       "200501BS00002                    1                        1   \n",
       "200501BS00003                    1                        1   \n",
       "200501BS00004                    1                        1   \n",
       "200501BS00005                    1                        2   \n",
       "\n",
       "                Special_Conditions_at_Site  Carriageway_Hazards  \\\n",
       "Accident_Index                                                    \n",
       "200501BS00001                            0                    0   \n",
       "200501BS00002                            0                    0   \n",
       "200501BS00003                            0                    0   \n",
       "200501BS00004                            0                    0   \n",
       "200501BS00005                            0                    0   \n",
       "\n",
       "                Urban_or_Rural_Area  \\\n",
       "Accident_Index                        \n",
       "200501BS00001                     1   \n",
       "200501BS00002                     1   \n",
       "200501BS00003                     1   \n",
       "200501BS00004                     1   \n",
       "200501BS00005                     1   \n",
       "\n",
       "                Did_Police_Officer_Attend_Scene_of_Accident  \\\n",
       "Accident_Index                                                \n",
       "200501BS00001                                             1   \n",
       "200501BS00002                                             1   \n",
       "200501BS00003                                             1   \n",
       "200501BS00004                                             1   \n",
       "200501BS00005                                             1   \n",
       "\n",
       "                LSOA_of_Accident_Location  \n",
       "Accident_Index                             \n",
       "200501BS00001                   E01002849  \n",
       "200501BS00002                   E01002909  \n",
       "200501BS00003                   E01002857  \n",
       "200501BS00004                   E01002840  \n",
       "200501BS00005                   E01002863  \n",
       "\n",
       "[5 rows x 31 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"accidents\")\n",
    "print(\"size=\",accidents.size)\n",
    "print(accidents.shape)\n",
    "accidents.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 352
    },
    "colab_type": "code",
    "id": "uC9FmdQ33ct5",
    "outputId": "e8f13886-ca84-4d16-dd94-64d98c13f577"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vehicles\n",
      "('size=', 4200000)\n",
      "(200000, 21)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Vehicle_Reference</th>\n",
       "      <th>Vehicle_Type</th>\n",
       "      <th>Towing_and_Articulation</th>\n",
       "      <th>Vehicle_Manoeuvre</th>\n",
       "      <th>Vehicle_Location-Restricted_Lane</th>\n",
       "      <th>Junction_Location</th>\n",
       "      <th>Skidding_and_Overturning</th>\n",
       "      <th>Hit_Object_in_Carriageway</th>\n",
       "      <th>Vehicle_Leaving_Carriageway</th>\n",
       "      <th>Hit_Object_off_Carriageway</th>\n",
       "      <th>...</th>\n",
       "      <th>Was_Vehicle_Left_Hand_Drive?</th>\n",
       "      <th>Journey_Purpose_of_Driver</th>\n",
       "      <th>Sex_of_Driver</th>\n",
       "      <th>Age_of_Driver</th>\n",
       "      <th>Age_Band_of_Driver</th>\n",
       "      <th>Engine_Capacity_(CC)</th>\n",
       "      <th>Propulsion_Code</th>\n",
       "      <th>Age_of_Vehicle</th>\n",
       "      <th>Driver_IMD_Decile</th>\n",
       "      <th>Driver_Home_Area_Type</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Accident_Index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200501BS00001</th>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>74</td>\n",
       "      <td>10</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00002</th>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>42</td>\n",
       "      <td>7</td>\n",
       "      <td>8268</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00003</th>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>35</td>\n",
       "      <td>6</td>\n",
       "      <td>8300</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00003</th>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>62</td>\n",
       "      <td>9</td>\n",
       "      <td>1762</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00004</th>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>49</td>\n",
       "      <td>8</td>\n",
       "      <td>1769</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Vehicle_Reference  Vehicle_Type  Towing_and_Articulation  \\\n",
       "Accident_Index                                                             \n",
       "200501BS00001                   1             9                        0   \n",
       "200501BS00002                   1            11                        0   \n",
       "200501BS00003                   1            11                        0   \n",
       "200501BS00003                   2             9                        0   \n",
       "200501BS00004                   1             9                        0   \n",
       "\n",
       "                Vehicle_Manoeuvre  Vehicle_Location-Restricted_Lane  \\\n",
       "Accident_Index                                                        \n",
       "200501BS00001                  18                                 0   \n",
       "200501BS00002                   4                                 0   \n",
       "200501BS00003                  17                                 0   \n",
       "200501BS00003                   2                                 0   \n",
       "200501BS00004                  18                                 0   \n",
       "\n",
       "                Junction_Location  Skidding_and_Overturning  \\\n",
       "Accident_Index                                                \n",
       "200501BS00001                   0                         0   \n",
       "200501BS00002                   3                         0   \n",
       "200501BS00003                   0                         0   \n",
       "200501BS00003                   0                         0   \n",
       "200501BS00004                   0                         0   \n",
       "\n",
       "                Hit_Object_in_Carriageway  Vehicle_Leaving_Carriageway  \\\n",
       "Accident_Index                                                           \n",
       "200501BS00001                           0                            0   \n",
       "200501BS00002                           0                            0   \n",
       "200501BS00003                           4                            0   \n",
       "200501BS00003                           0                            0   \n",
       "200501BS00004                           0                            0   \n",
       "\n",
       "                Hit_Object_off_Carriageway          ...            \\\n",
       "Accident_Index                                      ...             \n",
       "200501BS00001                            0          ...             \n",
       "200501BS00002                            0          ...             \n",
       "200501BS00003                            0          ...             \n",
       "200501BS00003                            0          ...             \n",
       "200501BS00004                            0          ...             \n",
       "\n",
       "                Was_Vehicle_Left_Hand_Drive?  Journey_Purpose_of_Driver  \\\n",
       "Accident_Index                                                            \n",
       "200501BS00001                              1                         15   \n",
       "200501BS00002                              1                          1   \n",
       "200501BS00003                              1                          1   \n",
       "200501BS00003                              1                         15   \n",
       "200501BS00004                              1                         15   \n",
       "\n",
       "                Sex_of_Driver  Age_of_Driver  Age_Band_of_Driver  \\\n",
       "Accident_Index                                                     \n",
       "200501BS00001               2             74                  10   \n",
       "200501BS00002               1             42                   7   \n",
       "200501BS00003               1             35                   6   \n",
       "200501BS00003               1             62                   9   \n",
       "200501BS00004               2             49                   8   \n",
       "\n",
       "                Engine_Capacity_(CC)  Propulsion_Code  Age_of_Vehicle  \\\n",
       "Accident_Index                                                          \n",
       "200501BS00001                     -1               -1              -1   \n",
       "200501BS00002                   8268                2               3   \n",
       "200501BS00003                   8300                2               5   \n",
       "200501BS00003                   1762                1               6   \n",
       "200501BS00004                   1769                1               4   \n",
       "\n",
       "                Driver_IMD_Decile  Driver_Home_Area_Type  \n",
       "Accident_Index                                            \n",
       "200501BS00001                   7                      1  \n",
       "200501BS00002                  -1                     -1  \n",
       "200501BS00003                   2                      1  \n",
       "200501BS00003                   1                      1  \n",
       "200501BS00004                   2                      1  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"vehicles\")\n",
    "print(\"size=\",vehicles.size)\n",
    "print(vehicles.shape)\n",
    "vehicles.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "colab_type": "code",
    "id": "zVQ_NruA3ct9",
    "outputId": "02cecc70-dd44-42be-9edb-30bd4f8815be"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "casualties\n",
      "('size=', 2800000)\n",
      "(200000, 14)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Vehicle_Reference</th>\n",
       "      <th>Casualty_Reference</th>\n",
       "      <th>Casualty_Class</th>\n",
       "      <th>Sex_of_Casualty</th>\n",
       "      <th>Age_of_Casualty</th>\n",
       "      <th>Age_Band_of_Casualty</th>\n",
       "      <th>Casualty_Severity</th>\n",
       "      <th>Pedestrian_Location</th>\n",
       "      <th>Pedestrian_Movement</th>\n",
       "      <th>Car_Passenger</th>\n",
       "      <th>Bus_or_Coach_Passenger</th>\n",
       "      <th>Pedestrian_Road_Maintenance_Worker</th>\n",
       "      <th>Casualty_Type</th>\n",
       "      <th>Casualty_Home_Area_Type</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Accident_Index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200501BS00001</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>37</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00002</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>37</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>-1</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00003</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>62</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00004</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>30</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00005</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>49</td>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>3</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Vehicle_Reference  Casualty_Reference  Casualty_Class  \\\n",
       "Accident_Index                                                          \n",
       "200501BS00001                   1                   1               3   \n",
       "200501BS00002                   1                   1               2   \n",
       "200501BS00003                   2                   1               1   \n",
       "200501BS00004                   1                   1               3   \n",
       "200501BS00005                   1                   1               1   \n",
       "\n",
       "                Sex_of_Casualty  Age_of_Casualty  Age_Band_of_Casualty  \\\n",
       "Accident_Index                                                           \n",
       "200501BS00001                 1               37                     7   \n",
       "200501BS00002                 1               37                     7   \n",
       "200501BS00003                 1               62                     9   \n",
       "200501BS00004                 1               30                     6   \n",
       "200501BS00005                 1               49                     8   \n",
       "\n",
       "                Casualty_Severity  Pedestrian_Location  Pedestrian_Movement  \\\n",
       "Accident_Index                                                                \n",
       "200501BS00001                   2                    1                    1   \n",
       "200501BS00002                   3                    0                    0   \n",
       "200501BS00003                   3                    0                    0   \n",
       "200501BS00004                   3                    5                    2   \n",
       "200501BS00005                   3                    0                    0   \n",
       "\n",
       "                Car_Passenger  Bus_or_Coach_Passenger  \\\n",
       "Accident_Index                                          \n",
       "200501BS00001               0                       0   \n",
       "200501BS00002               0                       4   \n",
       "200501BS00003               0                       0   \n",
       "200501BS00004               0                       0   \n",
       "200501BS00005               0                       0   \n",
       "\n",
       "                Pedestrian_Road_Maintenance_Worker  Casualty_Type  \\\n",
       "Accident_Index                                                      \n",
       "200501BS00001                                   -1              0   \n",
       "200501BS00002                                   -1             11   \n",
       "200501BS00003                                   -1              9   \n",
       "200501BS00004                                   -1              0   \n",
       "200501BS00005                                   -1              3   \n",
       "\n",
       "                Casualty_Home_Area_Type  \n",
       "Accident_Index                           \n",
       "200501BS00001                         1  \n",
       "200501BS00002                         1  \n",
       "200501BS00003                         1  \n",
       "200501BS00004                         1  \n",
       "200501BS00005                        -1  "
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"casualties\")\n",
    "print(\"size=\",casualties.size)\n",
    "print(casualties.shape)\n",
    "casualties.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "_uuid": "06b5266e977f07acd3ad0018bd698fce03a17cda",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "WFYs8fhM3cuB",
    "outputId": "90336b75-b590-4ace-bb82-6fd06c84dbd4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done joining\n",
      "(3144481, 52)\n"
     ]
    }
   ],
   "source": [
    "accidents = accidents.join(vehicles, how='outer')\n",
    "print(\"done joining\")\n",
    "print(accidents.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GCDV4MoHAW2b"
   },
   "source": [
    "#joining the tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "033c3bd61ac666e3fc87316f2642aff6dcab1719",
    "colab_type": "text",
    "id": "PCnEsvq93cuG"
   },
   "source": [
    "## Identifying Missing Values\n",
    "\n",
    "In this particular dataset, there are two types of missing values '-1' and 'Nan'. We will invesitigate each column with total missing values.\n",
    "We will not be imputing any mean or median value since the dataset is big enough to perform analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_uuid": "292691d5d1b20bf1664cb13dbda2c982f5563590",
    "colab": {},
    "colab_type": "code",
    "id": "R7vmqghW3cuH"
   },
   "outputs": [],
   "source": [
    "# accidents.drop(['Location_Easting_OSGR', 'Location_Northing_OSGR','LSOA_of_Accident_Location',\n",
    "#                 'Junction_Control' ,'2nd_Road_Class'], axis=1, inplace=True)\n",
    "\n",
    "#combining two columns\n",
    "accidents['Date_time'] = accidents['Date'] +' '+ accidents['Time']\n",
    "\n",
    "for col in accidents.columns:\n",
    "    accidents = (accidents[accidents[col]!=-1])\n",
    "    #print(col ,' ' , x)\n",
    "for col in casualties.columns:\n",
    "    casualties = (casualties[casualties[col]!=-1])\n",
    "\n",
    "accidents['Date_time'] = pd.to_datetime(accidents.Date_time)\n",
    "accidents.drop(['Date','Time'],axis =1 , inplace=True)\n",
    "accidents.dropna(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c010b516516f74ec01f0347bee12c2dfe11f8edd",
    "colab_type": "text",
    "id": "L9KBGPue3cuL"
   },
   "source": [
    "Our dataset is clean to do some analysis. We would be using very few columns to do analysis since the dataset is fairly large."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "190d4df6e3bfef24aab5176a410ae1890ef95108",
    "colab_type": "text",
    "id": "AUiA2nfH3cuM"
   },
   "source": [
    "# Data Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "9f62ed3d0e6aeb5dd14df572d477357978c58da2",
    "colab_type": "text",
    "id": "bi8b5fhL3cuN"
   },
   "source": [
    "#### The first thing we can do is to find out about accidents time to get intution and some driver's age who are involved in the accident.\n",
    "- We can find out the number of accidents on the days of a week.\n",
    "- We can find out about the accidents number using hours of the day.\n",
    "- Finding out about the age of driver can tell us more about the accidents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "210383c1cf401845104e000c33815343ed939d91",
    "colab": {},
    "colab_type": "code",
    "id": "h_KW4hsq3cuP"
   },
   "outputs": [],
   "source": [
    "# plt.figure(figsize=(12,6))\n",
    "# accidents.Date_time.dt.dayofweek.hist(bins=7,rwidth=0.55,alpha=0.5, color= 'orange')\n",
    "# plt.title('Accidents on the day of a week' , fontsize= 30)\n",
    "# plt.grid(False)\n",
    "# plt.ylabel('Accident count' , fontsize = 20)\n",
    "# plt.xlabel('0 - Sunday ,  1 - Monday  ,2 - Tuesday , 3 - Wednesday , 4 - Thursday , 5 - Friday , 6 - Saturday' , fontsize = 13)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "9a2e5d2440529e7a60e4e689dbf8fc7714a7d044",
    "colab_type": "text",
    "id": "VvwGqi7D3cuU"
   },
   "source": [
    "As we can see that thursday has the highest amount of accidents in this dataset from 2005 to 2015. We have to keep in mind that accidents numbers could be depending on traffic amount on particular day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "f2b4c64532386f9263b7f9bdd26cbac22ea8054a",
    "colab": {},
    "colab_type": "code",
    "id": "8Uu6iaFp3cuV"
   },
   "outputs": [],
   "source": [
    "# plt.figure(figsize=(12,6))\n",
    "# accidents.Date_time.dt.hour.hist(rwidth=0.75,alpha =0.50, color= 'orange')\n",
    "# plt.title('Time of the day/night',fontsize= 30)\n",
    "# plt.grid(False)\n",
    "# plt.xlabel('Time 0-23 hours' , fontsize = 20)\n",
    "# plt.ylabel('Accident count' , fontsize = 15)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "654850f800234a8784e81e447142b44c6b99d278",
    "colab_type": "text",
    "id": "bO-hPhdD3cuZ"
   },
   "source": [
    "We found out that the most of accidents happened around after noon. We can assume that this time of the day has the most traffic moving such as people leaving from work.\n",
    "\n",
    "\n",
    "#### Age band of casualities\n",
    "\n",
    "In this dataset, age band is grouped in 11 different codes. We will create the labels and pass it to the plot as xticks so we can have idea about the bins representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "0eb24ee1383bc02820ac4af76e07203b3cc96db3",
    "colab": {},
    "colab_type": "code",
    "id": "lcEbJdpJ3cua"
   },
   "outputs": [],
   "source": [
    "# objects = ['0','0-5','6-10','11-15','16-20','21-25','26-35',\n",
    "#           '36-45', '46-55','56-65','66-75','75+']\n",
    "\n",
    "# plt.figure(figsize=(12,6))\n",
    "# casualties.Age_Band_of_Casualty.hist(bins = 11,alpha=0.5,rwidth=0.90, color= 'red',)\n",
    "# plt.title('Age of people involved in the accidents', fontsize = 25)\n",
    "# plt.grid(False)\n",
    "# y_pos = np.arange(len(objects))\n",
    "# plt.xticks(y_pos , objects)\n",
    "# plt.ylabel('Accident count' , fontsize = 15)\n",
    "# plt.xlabel('Age of Drivers', fontsize = 15,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6ef96ceff3694498d0e12ed5c214519e3a9f56f3",
    "colab_type": "text",
    "id": "IXxJkD993cuc"
   },
   "source": [
    "This is very interesting fact about this dataset. Most of the drivers age is around 225 to 35 who are involved in the accident. However, we do not know the number of drivers with age 25 to 35 on the road compare to other ages. Intutively, I would assume that the driver with age 25 to 35 are more in the number of drivers with different age."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "465462cdac777a4ba6b1351808a1f81eb7658fca",
    "colab": {},
    "colab_type": "code",
    "id": "-1J2ZTYv3cud"
   },
   "outputs": [],
   "source": [
    "# speed_zone_accidents = accidents.loc[accidents['Speed_limit'].isin(['20' ,'30' ,'40' ,'50' ,'60' ,'70'])]\n",
    "# speed  = speed_zone_accidents.Speed_limit.value_counts()\n",
    "\n",
    "# explode = (0.0, 0.0, 0.0 , 0.0 ,0.0,0.0) \n",
    "# plt.figure(figsize=(10,8))\n",
    "# plt.pie(speed.values,  labels=None, \n",
    "#         autopct='%.1f',pctdistance=0.8, labeldistance=1.9 ,explode = explode, shadow=False, startangle=160,textprops={'fontsize': 15})\n",
    " \n",
    "# plt.axis('equal')\n",
    "# plt.legend(speed.index, bbox_to_anchor=(1,0.7), loc=\"center right\", fontsize=15, \n",
    "#            bbox_transform=plt.gcf().transFigure)\n",
    "# plt.figtext(.5,.9,'Accidents percentage in Speed Zone', fontsize=25, ha='center')\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "510d4d4fcf5618e3db3fea5fcd125dbe07d06d5d",
    "colab_type": "text",
    "id": "x-fYX0d33cuf"
   },
   "source": [
    "Most of the accidents occured on the road where the speed limit is 30. I was expecting more accidents on highway or major roadways. Some of the accidents could be cause of stop sign, changing lanes or turning into parking lot etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "cbbb531cc52aea344a7f0e34f7158feedcfecf05",
    "colab_type": "text",
    "id": "JCXnRJxN3cug"
   },
   "source": [
    "## Co-relation between variables\n",
    "\n",
    "Since our dataset is in numeric values. We can findout correlation between columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "741ba44e0e1154e24ae0b8a0d9ed2e42710bb5f5",
    "colab": {},
    "colab_type": "code",
    "id": "EWqSK0Iq3cuh"
   },
   "outputs": [],
   "source": [
    "# corr =  accidents.corr()\n",
    "# plt.subplots(figsize=(20,9))\n",
    "# sns.heatmap(corr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ebfdadae90f44fd7783071d7c895e3d492a0142e",
    "colab_type": "text",
    "id": "KJrHJtc63cuk"
   },
   "source": [
    "As we see that there is not so much strong correlations between any variables. I was expecting weather condition to be strong correlation with any of the variable. \n",
    "- There is only one postiive strong correlation between speed limit and Urban or Rural Area. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "996760b6548e85fa80d61fee849647190aa946de",
    "colab": {},
    "colab_type": "code",
    "id": "LSOy9if93cul"
   },
   "outputs": [],
   "source": [
    "# accidents_2014 = accidents[accidents.Date_time.dt.year ==2014]\n",
    "# accidents_2014_01 = accidents_2014[accidents_2014.Accident_Severity == 1]\n",
    "# accidents_2014_02 = accidents_2014[accidents_2014.Accident_Severity == 2]\n",
    "# accidents_2014_03 = accidents_2014[accidents_2014.Accident_Severity == 3]\n",
    "# print(\"done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e5ff74bdfa38e199c42538422ae98cf0cd0fb85c",
    "colab_type": "text",
    "id": "MazgZGbJ3cuo"
   },
   "source": [
    "##  Google Maps\n",
    "\n",
    "Plotting accidents Location on Google Maps\n",
    "Now we will be using google maps to plot the accidents. Using longitude and latitude information, we can see what area has the most accidents. However, it actually depends on how much traffic the area has. We can also get the idea of busiest area even if we do not want to look at just accidents. The accident plots acan give us really good idea about traffic in any area of the UK.\n",
    "\n",
    "Also, I have taken the screenshot of output plots so it can be seen when saved in html or pdf format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "fcb2b7d1da3990b459d2e25c4dfec7316e80d35a",
    "colab": {},
    "colab_type": "code",
    "id": "qmCINqi13cup"
   },
   "outputs": [],
   "source": [
    "# ! pip install gmaps\n",
    "# #!jupyter nbextension enable --py gmaps\n",
    "# import gmaps\n",
    "# from ipywidgets.embed import embed_minimal_html\n",
    "# gmaps.configure(api_key='AIzaSyDFOjxJ23DfYRLTqEuNsgnqwP0E79Aybpk')\n",
    "\n",
    "# fig = gmaps.figure(center=(53.0, 1.0), zoom_level=6)\n",
    "# heatmap_layer = gmaps.heatmap_layer(accidents_2014_01[[\"Latitude\", \"Longitude\"]],\n",
    "#                                     max_intensity=30,point_radius=5)\n",
    "# heatmap_layer = gmaps.heatmap_layer(accidents_2014_02[[\"Latitude\", \"Longitude\"]],\n",
    "#                                     max_intensity=5,point_radius=3)\n",
    "# heatmap_layer = gmaps.heatmap_layer(accidents_2014_03[[\"Latitude\", \"Longitude\"]],\n",
    "#                                     max_intensity=1,point_radius=1)\n",
    "# fig.add_layer(heatmap_layer)\n",
    "# fig\n",
    "# embed_minimal_html('export1.html', views=[fig])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "3236987c042659674fbca12d7faffa3f1dee1f8a",
    "colab": {},
    "colab_type": "code",
    "id": "2SC0cl_13cuu"
   },
   "outputs": [],
   "source": [
    "# import matplotlib.image as mpimg\n",
    "# plt.figure(figsize=(18,8))\n",
    "# img=mpimg.imread('../input/photos/map1.png')\n",
    "# imgplot = plt.imshow(img)\n",
    "# plt.grid(False)\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "7bef5b0f8884e9ffff4cec4fdbffda5d5f379d49",
    "colab": {},
    "colab_type": "code",
    "id": "H2iyZbzS3cuw"
   },
   "outputs": [],
   "source": [
    "# import gmaps\n",
    "# gmaps.configure(api_key=\"AIzaSyDFOjxJ23DfYRLTqEuNsgnqwP0E79Aybpk\") \n",
    "\n",
    "# maps_df = accidents_2014_01[['Latitude', 'Longitude']]\n",
    "# maps_layer = gmaps.symbol_layer(\n",
    "#    maps_df, fill_color=\"green\", stroke_color=\"red\", scale=1\n",
    "# )\n",
    "# fig = gmaps.figure()\n",
    "# fig.add_layer(maps_layer)\n",
    "# fig\n",
    "# print(\"done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "375e6dcffddbc0d6ca5e22cad1d2f0f06f345864",
    "colab": {},
    "colab_type": "code",
    "id": "j4_b6TU03cuy"
   },
   "outputs": [],
   "source": [
    "# import matplotlib.image as mpimg\n",
    "# plt.figure(figsize=(18,8))\n",
    "# img=mpimg.imread('../input/photos/map2.png')\n",
    "# imgplot = plt.imshow(img)\n",
    "# plt.grid(False)\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a8e320ab1318e7bb3a0bd1a64d3389dd9227ad22",
    "colab_type": "text",
    "id": "MISMvhcq3cu0"
   },
   "source": [
    "As we can see that most of fatal accidents happened locally within cities instead on highways. It could be the reason of the traffic is more congested locally than on highways."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "525e94433411de32b1c6b52f778abf07bc1f7883",
    "colab_type": "text",
    "id": "FLCysK8l3cu1"
   },
   "source": [
    "# Machine Learning\n",
    "\n",
    "We will be looking at different columns to figure out predicting about the accidents severity. After we can predict the accident severity, we can make some recommendation to law enforcement for looking into this and be prepared for the future. We can also have more emergency medical services available for those situations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "_uuid": "5626d86298cd35b6ce5367d1331a9dbf279b96fd",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "gpAn3Ab43cu2",
    "outputId": "e5423301-d880-4391-cdcc-7805b8cb8483"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.svm import SVC, LinearSVC\n",
    "from sklearn.metrics import log_loss\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6f440c02334d1e37d67a6b246effd17d4425de10",
    "colab_type": "text",
    "id": "_SG1CA2P3cu4"
   },
   "source": [
    "## Normalize the Data\n",
    "There are few columns that we will standarize, so it would not effect negatively on our machine learning algorithms. Age of driver is from 18 to 88 in the dataset and we can normalize it. Also, the age of vehicle is also from 0 to 100 and it can skew the performance of your machine learning algorithm and we will normalize this predictor too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "baabdc534c038c27b30123553e6b2ac2680879d4",
    "colab": {},
    "colab_type": "code",
    "id": "cWGLa9kz3cu5"
   },
   "outputs": [],
   "source": [
    "# sns.distplot(accidents['Age_of_Driver']);\n",
    "# fig = plt.figure()\n",
    "# sns.distplot(accidents['Age_of_Vehicle']);\n",
    "# fig = plt.figure()\n",
    "# print(\"done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "_uuid": "ac0701bcf6d5d46bc1804ca6a2065ffad1e574aa",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "kFs_8VnY3cu7",
    "outputId": "b24388e3-28a0-40aa-c2c0-97d17fefeb9d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done\n"
     ]
    }
   ],
   "source": [
    "\n",
    "accidents['Age_of_Driver'] = np.log(accidents['Age_of_Driver'])\n",
    "accidents['Age_of_Vehicle'] = np.log(accidents['Age_of_Vehicle'])\n",
    "# sns.distplot(accidents['Age_of_Driver']);\n",
    "# fig = plt.figure()\n",
    "# sns.distplot(accidents['Age_of_Vehicle']);\n",
    "# fig = plt.figure()\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Location_Easting_OSGR</th>\n",
       "      <th>Location_Northing_OSGR</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Police_Force</th>\n",
       "      <th>Accident_Severity</th>\n",
       "      <th>Number_of_Vehicles</th>\n",
       "      <th>Number_of_Casualties</th>\n",
       "      <th>Day_of_Week</th>\n",
       "      <th>Local_Authority_(District)</th>\n",
       "      <th>...</th>\n",
       "      <th>Journey_Purpose_of_Driver</th>\n",
       "      <th>Sex_of_Driver</th>\n",
       "      <th>Age_of_Driver</th>\n",
       "      <th>Age_Band_of_Driver</th>\n",
       "      <th>Engine_Capacity_(CC)</th>\n",
       "      <th>Propulsion_Code</th>\n",
       "      <th>Age_of_Vehicle</th>\n",
       "      <th>Driver_IMD_Decile</th>\n",
       "      <th>Driver_Home_Area_Type</th>\n",
       "      <th>Date_time</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Accident_Index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200501BS00007</th>\n",
       "      <td>524220.0</td>\n",
       "      <td>180830.0</td>\n",
       "      <td>-0.211277</td>\n",
       "      <td>51.512695</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>15.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.713572</td>\n",
       "      <td>7.0</td>\n",
       "      <td>4266.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.386294</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2005-01-13 20:40:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00010</th>\n",
       "      <td>527350.0</td>\n",
       "      <td>177650.0</td>\n",
       "      <td>-0.167342</td>\n",
       "      <td>51.483420</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>15.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.555348</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.564949</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2005-01-15 22:43:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00011</th>\n",
       "      <td>524550.0</td>\n",
       "      <td>180810.0</td>\n",
       "      <td>-0.206531</td>\n",
       "      <td>51.512443</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.737670</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8268.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.693147</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2005-01-15 16:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00012</th>\n",
       "      <td>526240.0</td>\n",
       "      <td>178900.0</td>\n",
       "      <td>-0.182872</td>\n",
       "      <td>51.494902</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>15.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.526361</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1988.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.791759</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2005-01-16 00:42:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00014</th>\n",
       "      <td>526170.0</td>\n",
       "      <td>177690.0</td>\n",
       "      <td>-0.184312</td>\n",
       "      <td>51.484044</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>15.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.944439</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1124.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.079442</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2005-01-25 20:48:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 51 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Location_Easting_OSGR  Location_Northing_OSGR  Longitude  \\\n",
       "Accident_Index                                                             \n",
       "200501BS00007                524220.0                180830.0  -0.211277   \n",
       "200501BS00010                527350.0                177650.0  -0.167342   \n",
       "200501BS00011                524550.0                180810.0  -0.206531   \n",
       "200501BS00012                526240.0                178900.0  -0.182872   \n",
       "200501BS00014                526170.0                177690.0  -0.184312   \n",
       "\n",
       "                 Latitude  Police_Force  Accident_Severity  \\\n",
       "Accident_Index                                               \n",
       "200501BS00007   51.512695             1                  3   \n",
       "200501BS00010   51.483420             1                  3   \n",
       "200501BS00011   51.512443             1                  3   \n",
       "200501BS00012   51.494902             1                  3   \n",
       "200501BS00014   51.484044             1                  3   \n",
       "\n",
       "                Number_of_Vehicles  Number_of_Casualties  Day_of_Week  \\\n",
       "Accident_Index                                                          \n",
       "200501BS00007                    2                     1            5   \n",
       "200501BS00010                    2                     2            7   \n",
       "200501BS00011                    2                     5            7   \n",
       "200501BS00012                    1                     1            1   \n",
       "200501BS00014                    2                     1            3   \n",
       "\n",
       "                Local_Authority_(District)         ...          \\\n",
       "Accident_Index                                     ...           \n",
       "200501BS00007                           12         ...           \n",
       "200501BS00010                           12         ...           \n",
       "200501BS00011                           12         ...           \n",
       "200501BS00012                           12         ...           \n",
       "200501BS00014                           12         ...           \n",
       "\n",
       "               Journey_Purpose_of_Driver  Sex_of_Driver  Age_of_Driver  \\\n",
       "Accident_Index                                                           \n",
       "200501BS00007                       15.0            1.0       3.713572   \n",
       "200501BS00010                       15.0            1.0       3.555348   \n",
       "200501BS00011                        1.0            1.0       3.737670   \n",
       "200501BS00012                       15.0            1.0       3.526361   \n",
       "200501BS00014                       15.0            2.0       2.944439   \n",
       "\n",
       "                Age_Band_of_Driver  Engine_Capacity_(CC)  Propulsion_Code  \\\n",
       "Accident_Index                                                              \n",
       "200501BS00007                  7.0                4266.0              1.0   \n",
       "200501BS00010                  6.0                1998.0              1.0   \n",
       "200501BS00011                  7.0                8268.0              2.0   \n",
       "200501BS00012                  6.0                1988.0              1.0   \n",
       "200501BS00014                  4.0                1124.0              1.0   \n",
       "\n",
       "                Age_of_Vehicle  Driver_IMD_Decile  Driver_Home_Area_Type  \\\n",
       "Accident_Index                                                             \n",
       "200501BS00007         1.386294                6.0                    1.0   \n",
       "200501BS00010         2.564949                3.0                    1.0   \n",
       "200501BS00011         0.693147                2.0                    1.0   \n",
       "200501BS00012         1.791759                4.0                    1.0   \n",
       "200501BS00014         2.079442                8.0                    1.0   \n",
       "\n",
       "                         Date_time  \n",
       "Accident_Index                      \n",
       "200501BS00007  2005-01-13 20:40:00  \n",
       "200501BS00010  2005-01-15 22:43:00  \n",
       "200501BS00011  2005-01-15 16:00:00  \n",
       "200501BS00012  2005-01-16 00:42:00  \n",
       "200501BS00014  2005-01-25 20:48:00  \n",
       "\n",
       "[5 rows x 51 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accidents.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8996af4753d28de07eb8b80690cf813b526ffa8e",
    "colab_type": "text",
    "id": "n3O6MbRD3cu9"
   },
   "source": [
    "## Spliting the data into training data and test data\n",
    "We will also consider few features as predictors for machine learning algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "4ff41e45b4ddcfbf1dc8e3741e5db4e4bd0188c6",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "NGoNxZ8V3cu-",
    "outputId": "845e621d-8197-456c-a5e1-fd6babcbc6e2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done\n"
     ]
    }
   ],
   "source": [
    "accident_ml = accidents.drop('Accident_Severity' ,axis=1)\n",
    "accident_ml = accident_ml[['Did_Police_Officer_Attend_Scene_of_Accident' , 'Age_of_Driver' ,'Vehicle_Type', 'Age_of_Vehicle','Engine_Capacity_(CC)','Day_of_Week' , 'Weather_Conditions' , 'Road_Surface_Conditions'\n",
    "                          , 'Light_Conditions', 'Sex_of_Driver' ,'Speed_limit']]\n",
    "\n",
    "accident_ml.head()\n",
    "\n",
    "# Split the data into a training and test set.\n",
    "X_train, X_test, y_train, y_test = train_test_split(accident_ml.values, \n",
    "                                              accidents['Accident_Severity'].values,test_size=0.20, random_state=99)\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "KC_qyd2-5bT5",
    "outputId": "d55078ec-8cdf-4f39-a4f7-171c986034df"
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'y_train' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-18-274fa6772014>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# y_train[100:200]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;31m# print(np.argmin(y_train))\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my_train\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m365\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m365\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'y_train' is not defined"
     ]
    }
   ],
   "source": [
    "# y_train[100:200]\n",
    "# print(np.argmin(y_train))\n",
    "print(y_train[365])\n",
    "print(X_train[365])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "id": "4kKsGBRzzFeD",
    "outputId": "40e17780-3cab-4357-ca6c-f2a04a3c1d17"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Did_Police_Officer_Attend_Scene_of_Accident</th>\n",
       "      <th>Age_of_Driver</th>\n",
       "      <th>Vehicle_Type</th>\n",
       "      <th>Age_of_Vehicle</th>\n",
       "      <th>Engine_Capacity_(CC)</th>\n",
       "      <th>Day_of_Week</th>\n",
       "      <th>Weather_Conditions</th>\n",
       "      <th>Road_Surface_Conditions</th>\n",
       "      <th>Light_Conditions</th>\n",
       "      <th>Sex_of_Driver</th>\n",
       "      <th>Speed_limit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Accident_Index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>200501BS00007</th>\n",
       "      <td>1</td>\n",
       "      <td>3.713572</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.386294</td>\n",
       "      <td>4266.0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00010</th>\n",
       "      <td>1</td>\n",
       "      <td>3.555348</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.564949</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00011</th>\n",
       "      <td>1</td>\n",
       "      <td>3.737670</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.693147</td>\n",
       "      <td>8268.0</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00012</th>\n",
       "      <td>1</td>\n",
       "      <td>3.526361</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.791759</td>\n",
       "      <td>1988.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200501BS00014</th>\n",
       "      <td>1</td>\n",
       "      <td>2.944439</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.079442</td>\n",
       "      <td>1124.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>2.0</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Did_Police_Officer_Attend_Scene_of_Accident  Age_of_Driver  \\\n",
       "Accident_Index                                                               \n",
       "200501BS00007                                             1       3.713572   \n",
       "200501BS00010                                             1       3.555348   \n",
       "200501BS00011                                             1       3.737670   \n",
       "200501BS00012                                             1       3.526361   \n",
       "200501BS00014                                             1       2.944439   \n",
       "\n",
       "                Vehicle_Type  Age_of_Vehicle  Engine_Capacity_(CC)  \\\n",
       "Accident_Index                                                       \n",
       "200501BS00007            9.0        1.386294                4266.0   \n",
       "200501BS00010            9.0        2.564949                1998.0   \n",
       "200501BS00011           11.0        0.693147                8268.0   \n",
       "200501BS00012            9.0        1.791759                1988.0   \n",
       "200501BS00014            9.0        2.079442                1124.0   \n",
       "\n",
       "                Day_of_Week  Weather_Conditions  Road_Surface_Conditions  \\\n",
       "Accident_Index                                                             \n",
       "200501BS00007             5                   1                        1   \n",
       "200501BS00010             7                   1                        1   \n",
       "200501BS00011             7                   1                        1   \n",
       "200501BS00012             1                   1                        1   \n",
       "200501BS00014             3                   1                        2   \n",
       "\n",
       "                Light_Conditions  Sex_of_Driver  Speed_limit  \n",
       "Accident_Index                                                \n",
       "200501BS00007                  4            1.0           30  \n",
       "200501BS00010                  4            1.0           30  \n",
       "200501BS00011                  1            1.0           30  \n",
       "200501BS00012                  4            1.0           30  \n",
       "200501BS00014                  4            2.0           30  "
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# X_train[]\n",
    "accident_ml.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "szOplie2xZsD",
    "outputId": "9cc52645-46dd-41cf-a614-c4b34a09d5e8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.00000000e+00, 3.61091791e+00, 9.00000000e+00, 2.39789527e+00,\n",
       "       2.98000000e+03, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,\n",
       "       1.00000000e+00, 2.00000000e+00, 3.00000000e+01])"
      ]
     },
     "execution_count": 17,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e88f3ee992a1e169244b33a4fd38968ed3b54433",
    "colab_type": "text",
    "id": "ROJkHrk53cvA"
   },
   "source": [
    "## Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "ddd13f370e56bbde94e945aa600b6feaedf3ffeb",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "nJUWfGm23cvA",
    "outputId": "0e9e57bf-ae01-424a-ac17-8c5ea2462ff6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Accuracy', 88.97)\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1   0.400000  0.016807  0.032258       119\n",
      "           2   0.190141  0.023356  0.041602      1156\n",
      "           3   0.898131  0.990104  0.941878     11015\n",
      "\n",
      "   micro avg   0.889748  0.889748  0.889748     12290\n",
      "   macro avg   0.496090  0.343423  0.338579     12290\n",
      "weighted avg   0.826714  0.889748  0.848390     12290\n",
      "\n",
      "done\n"
     ]
    }
   ],
   "source": [
    "random_forest = RandomForestClassifier(n_estimators=200)\n",
    "random_forest.fit(X_train,y_train)\n",
    "Y_pred = random_forest.predict(X_test)\n",
    "random_forest.score(X_test, y_test)\n",
    "acc_random_forest1 = round(random_forest.score(X_test, y_test) * 100, 2)\n",
    "\n",
    "sk_report = classification_report(\n",
    "    digits=6,\n",
    "    y_true=y_test, \n",
    "    y_pred=Y_pred)\n",
    "print(\"Accuracy\" , acc_random_forest1)\n",
    "print(sk_report)\n",
    "pd.crosstab(y_test, Y_pred, rownames=['Actual'], colnames=['Predicted'], margins=True)\n",
    "\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "s5Klt0m_5Fca",
    "outputId": "cb2e8f35-0e14-4fa4-8f04-9387ed0fb264"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Accuracy', 88.97)\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           1   0.400000  0.016807  0.032258       119\n",
      "           2   0.190141  0.023356  0.041602      1156\n",
      "           3   0.898131  0.990104  0.941878     11015\n",
      "\n",
      "   micro avg   0.889748  0.889748  0.889748     12290\n",
      "   macro avg   0.496090  0.343423  0.338579     12290\n",
      "weighted avg   0.826714  0.889748  0.848390     12290\n",
      "\n",
      "done\n"
     ]
    }
   ],
   "source": [
    "sk_report = classification_report(\n",
    "    digits=6,\n",
    "    y_true=y_test, \n",
    "    y_pred=Y_pred)\n",
    "print(\"Accuracy\" , acc_random_forest1)\n",
    "print(sk_report)\n",
    "pd.crosstab(y_test, Y_pred, rownames=['Actual'], colnames=['Predicted'], margins=True)\n",
    "\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 657
    },
    "colab_type": "code",
    "id": "K17fclfH-Yek",
    "outputId": "282f1f41-3bd6-4927-f4bc-339d8271bb66"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                Did_Police_Officer_Attend_Scene_of_Accident  Age_of_Driver  \\\n",
      "Accident_Index                                                               \n",
      "200501BS00007                                             1       3.713572   \n",
      "200501BS00010                                             1       3.555348   \n",
      "200501BS00011                                             1       3.737670   \n",
      "200501BS00012                                             1       3.526361   \n",
      "200501BS00014                                             1       2.944439   \n",
      "\n",
      "                Vehicle_Type  Age_of_Vehicle  Engine_Capacity_(CC)  \\\n",
      "Accident_Index                                                       \n",
      "200501BS00007            9.0        1.386294                4266.0   \n",
      "200501BS00010            9.0        2.564949                1998.0   \n",
      "200501BS00011           11.0        0.693147                8268.0   \n",
      "200501BS00012            9.0        1.791759                1988.0   \n",
      "200501BS00014            9.0        2.079442                1124.0   \n",
      "\n",
      "                Day_of_Week  Weather_Conditions  Road_Surface_Conditions  \\\n",
      "Accident_Index                                                             \n",
      "200501BS00007             5                   1                        1   \n",
      "200501BS00010             7                   1                        1   \n",
      "200501BS00011             7                   1                        1   \n",
      "200501BS00012             1                   1                        1   \n",
      "200501BS00014             3                   1                        2   \n",
      "\n",
      "                Light_Conditions  Sex_of_Driver  Speed_limit  \n",
      "Accident_Index                                                \n",
      "200501BS00007                  4            1.0           30  \n",
      "200501BS00010                  4            1.0           30  \n",
      "200501BS00011                  1            1.0           30  \n",
      "200501BS00012                  4            1.0           30  \n",
      "200501BS00014                  4            2.0           30  \n",
      "(56, 11)\n",
      "[  1.           3.73766962   3.           0.69314718 125.\n",
      "   4.           1.           1.           1.           1.\n",
      "  30.        ]\n",
      "done\n"
     ]
    }
   ],
   "source": [
    "#Predict\n",
    "# sample = [7.0,3.2,4.7,1.4]\n",
    "# print(\"done\")\n",
    "# sample.reshape(1, -1)\n",
    "# result = clf.predict(sample).reshape(1, -1)\n",
    "# result.reshape(1,-1)\n",
    "\n",
    "\n",
    "\n",
    "print(accident_ml.head())\n",
    "print(X_train.shape)\n",
    "print(X_train[0])\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 181
    },
    "colab_type": "code",
    "id": "4nYTZ1xr-Rby",
    "outputId": "f12ab6c2-ad3e-47ef-ca95-de8f7aa1fe1d"
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m\u001b[0m",
      "\u001b[0;31mNameError\u001b[0mTraceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-dd2fe0e9b64c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mY_pred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrandom_forest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m365\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreshape\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mY_pred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'random_forest' is not defined"
     ]
    }
   ],
   "source": [
    "Y_pred = random_forest.predict(X_test[365].reshape(1, -1))\n",
    "print(Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 250
    },
    "colab_type": "code",
    "id": "Mnj7htHS8-QG",
    "outputId": "9f97e6b3-761f-4258-95b7-991e34f84676"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "E: Package 'python-software-properties' has no installation candidate\n",
      "Selecting previously unselected package google-drive-ocamlfuse.\n",
      "(Reading database ... 131323 files and directories currently installed.)\n",
      "Preparing to unpack .../google-drive-ocamlfuse_0.7.1-0ubuntu3~ubuntu18.04.1_amd64.deb ...\n",
      "Unpacking google-drive-ocamlfuse (0.7.1-0ubuntu3~ubuntu18.04.1) ...\n",
      "Setting up google-drive-ocamlfuse (0.7.1-0ubuntu3~ubuntu18.04.1) ...\n",
      "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n",
      "Please, open the following URL in a web browser: https://accounts.google.com/o/oauth2/auth?client_id=32555940559.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&response_type=code&access_type=offline&approval_prompt=force\n",
      "··········\n",
      "Please, open the following URL in a web browser: https://accounts.google.com/o/oauth2/auth?client_id=32555940559.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive&response_type=code&access_type=offline&approval_prompt=force\n",
      "Please enter the verification code: Access token retrieved correctly.\n",
      "done connecting to google drive\n"
     ]
    }
   ],
   "source": [
    "#connecting to GOOGLE DRIVE and saving the model \n",
    "!apt-get install -y -qq software-properties-common python-software-properties module-init-tools\n",
    "!add-apt-repository -y ppa:alessandro-strada/ppa 2>&1 > /dev/null\n",
    "!apt-get update -qq 2>&1 > /dev/null\n",
    "!apt-get -y install -qq google-drive-ocamlfuse fuse\n",
    "from google.colab import auth\n",
    "auth.authenticate_user()\n",
    "from oauth2client.client import GoogleCredentials\n",
    "creds = GoogleCredentials.get_application_default()\n",
    "import getpass\n",
    "!google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret} < /dev/null 2>&1 | grep URL\n",
    "vcode = getpass.getpass()\n",
    "!echo {vcode} | google-drive-ocamlfuse -headless -id={creds.client_id} -secret={creds.client_secret}\n",
    "!mkdir -p drive\n",
    "!google-drive-ocamlfuse drive\n",
    "\n",
    "\n",
    "print(\"done connecting to google drive\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "h12IwQW6_M6l",
    "outputId": "a297da23-57dd-4ce4-9aaf-0385b194675e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accidents.csv  adc.json  casualties.csv  drive\tsample_data  vehicles.csv\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "qHq1fxhy_UcA",
    "outputId": "3e689072-266e-429b-8efb-8f47378b7e80"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['drive/litemodel.sav']"
      ]
     },
     "execution_count": 27,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.externals import joblib\n",
    "modelfile=\"drive/litemodel.sav\"\n",
    "joblib.dump(random_forest,modelfile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "colab_type": "code",
    "id": "6A1OipWHAQ8d",
    "outputId": "3ffbcac6-c92c-4ebb-fa25-8c90a6a6f285"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "loaded model\n"
     ]
    }
   ],
   "source": [
    "# load the model from drive\n",
    "loaded_model= joblib.load(modelfile)\n",
    "# result=loaded_model.score(X_test, y_test)\n",
    "# print(result) \n",
    "loaded_model\n",
    "print(\"loaded model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 169
    },
    "colab_type": "code",
    "id": "TqRGaAmVzZpR",
    "outputId": "c6e21eec-76c3-44f2-fd46-d263a96d628c"
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m\u001b[0m",
      "\u001b[0;31mNameError\u001b[0mTraceback (most recent call last)",
      "\u001b[0;32m<ipython-input-7-73020798007e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mX_train\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'X_train' is not defined"
     ]
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zPVB3scDzfCR"
   },
   "outputs": [],
   "source": [
    "X_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "0Bk5UhIEt0tA",
    "outputId": "d9ec7745-01d4-4ccb-b375-b3baec39d921"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\n"
     ]
    }
   ],
   "source": [
    "X= [1.00000000e+00,3.17805383e+00 , 9.00000000e+00 , 2.70805020e+00,1.67900000e+03,6.00000000e+00,1.00000000e+00,1.00000000e+00,1.00000000e+00,1.00000000e+00,3.00000000e+01]\n",
    "# X= [3.5,11,1.6,8300.0,5,1,1,1,1,30]\n",
    "X= np.array([  1.        ,   3.73766962,   3.        ,   0.69314718,\n",
    "       125.        ,   4.        ,   1.        ,   1.        ,\n",
    "         1.        ,   1.        ,  30.        ])\n",
    "# Y = loaded_model.predict(X_train[0].reshape(1, -1))\n",
    "Y = loaded_model.predict(X.reshape(1, -1))\n",
    "print(Y) #printed the "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 133
    },
    "colab_type": "code",
    "id": "6UWvHZjZ-ZdL",
    "outputId": "d3262bb8-dfb1-4da6-86b6-4e9f0481f4ca"
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-5-3c2c122153b4>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    X=[200501BS00003,\u001b[0m\n\u001b[0m                   ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "#Check the result\n",
    "# result[0]\n",
    "# X=[200501BS00003, \n",
    "# 3.555348,  \n",
    "# 11.0, \n",
    "# 1.609438, \n",
    "# 8300.0,\n",
    "# 5,         \n",
    "# 1,  # Weather_Conditions =int\n",
    "# 1,  #Road_Surface_Conditions=int\n",
    "# 1,           #Light_Conditions=int\n",
    "# 1.0,            #Sex_of_Driver=float\n",
    "# 30]            #Speed_limit=int\n",
    "# Y = loaded_model.predict(X)\n",
    "# print(Y)\n",
    "# # loaded_model.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Xs9vsfYn-cDF"
   },
   "outputs": [],
   "source": [
    "# !pip install azureml\n",
    "\n",
    "from azureml import services\n",
    "@services.publish('f8311c4e9dd942c4b5fb2b322c164a59', 'tk6XlAPlYCw+cAbzvQsMREYwgR6OHrY4o/1Xjg82Rqlt+aHo89SXHtLseUc0Dn3VYrQzl+3q8UTzIgnw5b36EA==')\n",
    "@services.types(Accident_Index = int, Age_of_Driver = float, Vehicle_Type=float , \n",
    "                Age_of_Vehicle = float, Engine_Capacity_CC = float, Day_of_Week = int,\n",
    "                Weather_Conditions =int, Road_Surface_Conditions=int,Light_Conditions=int, Sex_of_Driver=float, Speed_limit=int)\n",
    "@services.returns(int) \n",
    "# 0,or 1,or 2\n",
    "\n",
    "def predictAccident2(Accident_Index, Age_of_Driver, Vehicle_Type, Age_of_Vehicle, Engine_Capacity_CC, Day_of_Week, Weather_Conditions, Road_Surface_Conditions, Light_Conditions,Sex_of_Driver, Speed_limit):\n",
    " inputArray = [Accident_Index, Age_of_Driver, Vehicle_Type, Age_of_Vehicle, Engine_Capacity_CC, Day_of_Week, Weather_Conditions, Road_Surface_Conditions, Light_Conditions, Sex_of_Driver, Speed_limit]\n",
    " Re = random_forest.predict(inputArray)\n",
    " return Re[0] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "o5HR73Zszuiw"
   },
   "outputs": [],
   "source": [
    "dir(predictAccident2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zBrM1zOE-g2z"
   },
   "outputs": [],
   "source": [
    "# @services.types(Accident_Index = int, Age_of_Driver = float, Vehicle_Type=float , \n",
    "#                 Age_of_Vehicle = float, Engine_Capacity_CC = float, Day_of_Week = int,\n",
    "#                 Weather_Conditions =int, Road_Surface_Conditions=int,Light_Conditions=int, Sex_of_Driver=float, Speed_limit=int)\n",
    "\n",
    "# predictAccident2.service( 200501BS00003 ,   \n",
    "#                          3.555348,  #Age_of_Driver = float\n",
    "#                          11.0,  #Vehicle_Type=float\n",
    "#                          1.609438, #Age_of_Vehicle = float,\n",
    "#                          8300.0, # Engine_Capacity_CC = float\n",
    "#                          5,          #Day_of_Week = int  \n",
    "#                          1,  # Weather_Conditions =int\n",
    "#                          1,  #Road_Surface_Conditions=int\n",
    "#                          1,           #Light_Conditions=int\n",
    "#                          1.0,            #Sex_of_Driver=float\n",
    "#                          30)             #Speed_limit=int"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oJ2nK6VY-iY4"
   },
   "outputs": [],
   "source": [
    "# predictAccident.service.help_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AOhU3O5_-leR"
   },
   "outputs": [],
   "source": [
    "# predictAccident.service.url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZLmPclDc-nZP"
   },
   "outputs": [],
   "source": [
    "# predictAccident.service.api_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ae20ccb692186d35f9f574b66c3265025d25e13c",
    "colab_type": "text",
    "id": "I2JLLhvo3cvC"
   },
   "source": [
    "## Logistic Regression "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "96978ce02fb5095b38972f85042ec4e66dce8d38",
    "colab": {},
    "colab_type": "code",
    "id": "hOKPTOgg3cvD"
   },
   "outputs": [],
   "source": [
    "# lr = LogisticRegression()\n",
    "# # Fit the model on the trainng data.\n",
    "# lr.fit(X_train, y_train)\n",
    "# y_pred = lr.predict(X_test)\n",
    "# sk_report = classification_report(\n",
    "#     digits=6,\n",
    "#     y_true=y_test, \n",
    "#     y_pred=y_pred)\n",
    "# print(\"Accuracy\", round(accuracy_score(y_pred, y_test)*100,2))\n",
    "# print(sk_report)\n",
    "# pd.crosstab(y_test, y_pred, rownames=['Actual'], colnames=['Predicted'], margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "714781275052e5512f4f6fb654fc006c322fde65",
    "colab_type": "text",
    "id": "0wExv_rP3cvF"
   },
   "source": [
    "## Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "b66350c57786ddac65f80ec4682ee28bf68250cb",
    "colab": {},
    "colab_type": "code",
    "id": "1EenDbbn3cvG"
   },
   "outputs": [],
   "source": [
    "# decision_tree = DecisionTreeClassifier()\n",
    "# decision_tree.fit(X_train, y_train)\n",
    "# Y_pred = decision_tree.predict(X_test)\n",
    "# acc_decision_tree1 = round(decision_tree.score(X_test, y_test) * 100, 2)\n",
    "# sk_report = classification_report(\n",
    "#     digits=6,\n",
    "#     y_true=y_test, \n",
    "#     y_pred=Y_pred)\n",
    "# print(\"Accuracy\", acc_decision_tree1)\n",
    "# print(sk_report)\n",
    "# ### Confusion Matrix \n",
    "# pd.crosstab(y_test, Y_pred, rownames=['Actual'], colnames=['Predicted'], margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "be6bd6b139f10cbf1434aee4f3ecb7e58ac14fee",
    "colab_type": "text",
    "id": "tQLaUl8E3cvJ"
   },
   "source": [
    "As we can see that Logistic regression did pretty well in terms of number. If we look carefully at the confusion matrix. We can definitely tell that Decision tree algorithm did much better. It predicted more fatal and serious injuries as true positive. The accuracy score is lower compare to another algorithm because other algorithm predicted majority of slightly accidents and those numbers are really high overall in the dataset. Confusion matrix helps us to understand what algorithm actually worked better in terms of looking at all different prediction of each class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "20e2285ebc104e46ead88fdf348d3c729106999f",
    "colab_type": "text",
    "id": "DDdEXjpj3cvJ"
   },
   "source": [
    "# Hyperparameters tuning for the models\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5bdaee73e2ed61b9649792f057c9fc05e2b30108",
    "colab_type": "text",
    "id": "mf-dPXWJ3cvK"
   },
   "source": [
    "### Logistic Regression with Hyperparameter tuning\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "422c67476c024b857edf2ba9cffb4108bba4d4e9",
    "colab": {},
    "colab_type": "code",
    "id": "7S91jzGr3cvN"
   },
   "outputs": [],
   "source": [
    "# from sklearn.linear_model import LogisticRegressionCV\n",
    "# lr = LogisticRegressionCV(cv=3, random_state=0,multi_class='multinomial')\n",
    "# # Fit the model on the trainng data.\n",
    "# lr.fit(X_train, y_train)\n",
    "# y_pred = lr.predict(X_test)\n",
    "# sk_report = classification_report(\n",
    "#     digits=6,\n",
    "#     y_true=y_test, \n",
    "#     y_pred=y_pred)\n",
    "# print(\"Accuracy\", round(accuracy_score(y_pred, y_test)*100,2))\n",
    "# print(sk_report)\n",
    "# pd.crosstab(y_test, y_pred, rownames=['Actual'], colnames=['Predicted'], margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "95f441cfef4974bc8a44bdbf07d7b8a85bc362ad",
    "colab_type": "text",
    "id": "90aA7t4m3cvR"
   },
   "source": [
    "As we can see that Logistic regression still didn't predict two classes of accident severity out of 3. Even though it is showing the 86.2% accuracy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5365138a8ce876cd45b43a05281cdb863424bad8",
    "colab_type": "text",
    "id": "DbLJ_Tr73cvS"
   },
   "source": [
    "### Decision Tree hyperparameters tuning\n",
    "\n",
    "All we are going to do is find the best values for mininum sample leaf and maximum features to get the best score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "c87f509414eac285236e01fca0ecd8a99d2bdbf8",
    "colab": {},
    "colab_type": "code",
    "id": "mv12EH0U3cvT"
   },
   "outputs": [],
   "source": [
    "# decision_tree = DecisionTreeClassifier(min_samples_leaf=12, max_features=4)\n",
    "# decision_tree.fit(X_train, y_train)\n",
    "# Y_pred = decision_tree.predict(X_test)\n",
    "# acc_decision_tree1 = round(decision_tree.score(X_test, y_test) * 100, 2)\n",
    "# sk_report = classification_report(\n",
    "#     digits=6,\n",
    "#     y_true=y_test, \n",
    "#     y_pred=Y_pred)\n",
    "# print(\"Accuracy\", acc_decision_tree1)\n",
    "# print(sk_report)\n",
    "# ### Confusion Matrix \n",
    "# pd.crosstab(y_test, Y_pred, rownames=['Actual'], colnames=['Predicted'], margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "e9052958b1ac2236b1789355a5a11a750f8e4aa1",
    "colab_type": "text",
    "id": "OAn4B8jd3cvX"
   },
   "source": [
    "We really didn't see much difference in Accident severity 1 and 2. However we did improve the accuracy of Accident severity 3. It jumped the accuracy from 75.1% to 85.8%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "04bc793b2310573e363f2e611cf256373afc6e0f",
    "colab_type": "text",
    "id": "SrEei6d03cvZ"
   },
   "source": [
    "###  Random Forest Hyperparameter tuning\n",
    "First, we will see the default parameters of the random forest model before we tune the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "382bebf8225802f780d951100eeac8b604f7e928",
    "colab": {},
    "colab_type": "code",
    "id": "GblkKqI13cva"
   },
   "outputs": [],
   "source": [
    "# random_forest.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1a759ccb8a633efec86563b508d9ac08657f87b1",
    "colab_type": "text",
    "id": "jhc1EEAO3cvd"
   },
   "source": [
    "We will implement the grid search using sklearn library. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "5c923e61216c5f4a70177cb46e748ab9fb976765",
    "colab": {},
    "colab_type": "code",
    "id": "6u8VZDfg3cvd"
   },
   "outputs": [],
   "source": [
    "# from sklearn.model_selection import RandomizedSearchCV\n",
    "# param_grid = {\n",
    "#     'bootstrap': [True],\n",
    "#     'max_depth': [80, 90, 100, 110],\n",
    "#     'max_features': [4, 5],\n",
    "#     'min_samples_leaf': [5, 10, 15],\n",
    "#     'min_samples_split': [8, 10, 12],\n",
    "#     'n_estimators': [100, 200, 300]\n",
    "# }\n",
    "# # Create a based model\n",
    "# random_f = RandomForestClassifier()\n",
    "# # Instantiate the grid search model\n",
    "# grid_search = RandomizedSearchCV(estimator = random_f, param_distributions = param_grid, \n",
    "#                           cv = 3, n_jobs = -1, verbose = 2)\n",
    "\n",
    "# grid_search.fit(X_train,y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7f9440cd53d5f18ef91c80e19fd7593d1a7590f4",
    "colab_type": "text",
    "id": "8aE7VbYK3cvf"
   },
   "source": [
    "## Feature importance\n",
    "We can use Sklearn's random forest library to find out the most important features. We will be plotting in  ascending order so we know what features are most important to predict the accident severity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "8fcb8231e0a2baf72e89f5985c5f20f05da9382c",
    "colab": {},
    "colab_type": "code",
    "id": "zoD0g_Su3cvg"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,6))\n",
    "feat_importances = pd.Series(random_forest.feature_importances_, index=accident_ml.columns)\n",
    "feat_importances.nlargest(5).plot(kind='barh')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "_uuid": "8a6e8de82ae44d10b8c2ccb687bba2c947be8f95",
    "colab": {},
    "colab_type": "code",
    "id": "VXeh19Yh3cvh"
   },
   "outputs": [],
   "source": [
    "# Y_pred = grid_search.predict(X_test)\n",
    "# acc_random_forest1 = round(grid_search.score(X_test, y_test) * 100, 2)\n",
    "\n",
    "# sk_report = classification_report(\n",
    "#     digits=6,\n",
    "#     y_true=y_test, \n",
    "#     y_pred=Y_pred)\n",
    "# print(\"Accuracy\" , acc_random_forest1)\n",
    "# print(sk_report)\n",
    "# pd.crosstab(y_test, Y_pred, rownames=['Actual'], colnames=['Predicted'], margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "bfceeaaa72237771033a5fb480018723e2011192",
    "colab_type": "text",
    "id": "zgrfAQmo3cvj"
   },
   "source": [
    "Random forest took lots of time to tune the hyperparameter. Most of the algorithm works well only with default values except decision tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1dc94a15c09d173b17e5e890d27355884a588ffb",
    "colab_type": "text",
    "id": "9Z7Vo6Hn3cvk"
   },
   "source": [
    "## Conclusion\n",
    "As we have implemented the Logistic Regression, Decision Tree and Random Forest algorithms to predict the accident severity. There are two things that we can conclude from this learning.\n",
    "\n",
    "#### Machine Learning Conclusion\n",
    "As we have tried three different algorithms to predict the accident severity. It was clear that Decision tree and Random Forest performed much better in terms of predicting all the classes of accident severity. Logistic regression has better accuracy but it does not mean it did better than other algorithm. We even tried multi-nomial to predict all the classes in hyperparameter tuning section. It still predicted only one of the higher occuring class.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "AUiA2nfH3cuM",
    "bO-hPhdD3cuZ",
    "I2JLLhvo3cvC",
    "0wExv_rP3cvF",
    "DDdEXjpj3cvJ",
    "mf-dPXWJ3cvK",
    "DbLJ_Tr73cvS",
    "SrEei6d03cvZ"
   ],
   "name": "colabazuremlFULL-traffic-accidents.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}