Project/finalproject.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../dsi.png\" style=\"height:128px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Project: Crime in India"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations on reaching the final project stage of DSI! As your final project today, we want you to look at data from India about crimes committed against women. Safety of women is a critical issue across India right now and we want you to take a data-centric approach to begin unpacking this topic. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "matplotlib.use('Agg')\n",
    "from datascience import Table, predicates\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "plt.style.use('fivethirtyeight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's make a table of crimes committed in each state in 2012."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "crime_data = Table.read_table('crime_by_state.csv')\n",
    "crime_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1: Warm-Up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A.** How many rows are there in this table? Write a line of code that tells us the number of rows in the table. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**B.** Find the population of all the states reported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "total_pop = sum(crime_data.____(_____))\n",
    "total_pop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**C.** Find the total amount of arrests in 2012."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "total_arrests = \n",
    "total_arrests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**D.** Calculate the total amount of arrests per hundred thousand people in 2012."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "10000 * _____ / ______"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E.** Do you think it’s a high fraction of the population? Consider the entire population of India. How would this reflect if the rate were applied to the whole country?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer Here** (double click to edit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2: Visualizing our Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A.** Before we do this, we need to add one column to our table. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the weighted crime by state (crimes per 100,000) using two columns from the table, and add this information as a new column to the crime_data table as a column called `\"Weighted Crime\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "total_crime_by_state = \n",
    "population_by_state = \n",
    "weighted_crime_by_state = np.divide(100000.0 * total_crime_by_state, population_by_state)\n",
    "crime_data = crime_data.with_column(\"________\", _________)\n",
    "crime_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**B.** Make a plot of the total male count and total female count of arrested peoples by state below. \n",
    "\n",
    "Hint: Put the names of the columns representing male and female arrests in the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "crime_data.____(\"_____\", np.array([\"____\", \"______\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**C.** Now we want to look at the *normalized* crime per state (per 100,000 people). This will let us look at how the states rank in terms of crimes relative to their populations. Make this plot below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**D.** Do the states rank similarly? Do they seem to follow a similar pattern in terms of number of crimes?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3: Convictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have looked at some raw crime data, let’s think about its consequences. Is any action being taken against offenders? What kind of data do you think you could look at that would help you answer this question?\n",
    "\n",
    "Let’s try looking at the number of persons convicted and corresponding numbers of acquittals and convictions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "conviction_data = Table.read_table(\"convictions_by_state.csv\")\n",
    "conviction_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A.** \n",
    "First, let's calculate the percentage of *arrests* that were *aquitted* and add this information as a new column to the crime_data table as a column called \"Percentage Acquitted\". Recall how you did this for the crime_data table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "number_acquitted = \n",
    "number_arrested = \n",
    "percentage_acquitted = (_______ / _______) * ______\n",
    "conviction_data = conviction_data.with_column(\"__________\", ___________)\n",
    "conviction_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**B.** Conduct your own calculations and make appropriate plots to try to answer the questions we have posed. Discuss your approach and potential results with your peers and instructor. We have left some blank space below for you to do this work.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Caution! Do you think looking at the acquittal rate was a fair way to assess action being taken against offenders? What does an acquittal really mean? We must understand the difference between an acquittal and a false case - no matter the crime. An acquittal occurs when there is not enough proof against the accused - not necessarily when the accuser is determined to be a liar. In fact, the acquittal rate for all crimes is approximately the same. The rate of acquittal for attempted murder for example is 73.4%. That doesn’t mean there is a false murder case epidemic! For rape it is 72.9%. So perhaps we should not have looked at the acquittal rate data to assess the conviction rate. What other types of data do you think we could have looked at to help us?*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3: Literacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the literacy data as a table from the raw literacy csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "literacy_data = Table.read_table(\"literacy_by_state.csv\")\n",
    "literacy_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A.** Add two columns to your `literacy_data` table called `Percentage Acquitted` and `Weighted Crime`. Use the `Percentage Acquitted` column from your `conviction_data` table and the `Weighted Crime` column from your `crime_data` table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "literacy_data = \n",
    "literacy_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**B.** Make a scatter plot comparing literacy rates in each state to the percentage of people acquitted. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "literacy_data.scatter(\"_______\", \"___________\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**C.** Calculate the correlation coefficient between these two factors. Use the `standard_units` function to help you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Calculating distance from the mean, dividing by standard deviation\n",
    "def standard_units(nums):\n",
    "    return (nums - np.mean(nums))/np.std(nums)\n",
    "#Average of the product of x and y values in standard units, takes in a table and strings representing two column names.\n",
    "def correlation(tbl, col_1, col_2):\n",
    "    return np.mean(standard_units(tbl.column(col_1)) * standard_units(tbl.column(col_2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "correlation(literacy_data, \"____\", \"_____\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**D.** What results did you find? Are they correlated? Is it a strong correlation? Even if they are correlated, does that say anything about causation?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your Answer Here**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**E.** Bonus– Plot predictions generated by the regression line (Use the `minimize` function for least-squares).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4: Qualitative Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Discuss these questions with your peers and the instructor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A.** What can you take away from the calculations and visualizations you have made today?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your Answer**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**B.** Where do you think there are potential holes in the data and in the calculations we have made? \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your Answer**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**C.** According to *The Hindu* newspaper, marital and other rape is grossly underreported in India.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"hindu_img.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why would there be motivation to keep the reported crime rate low? Who is held accountable for these crime rates? Politicians, police, citizens?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your Answer**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Today, wewere able to see one of the many uses of data in today's world and connected our knowledge of data science with an important issue in society. You'll be able to use the material from the course *anywhere* and *anytime* in life, and we hope you learned a lot about how to look at the world and use numbers to describe it, computers to process it, and your brain to draw conclusions!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}