MinnPost Crime

An application that explores crime in Minnesota, specifically Minneapolis. The live application can be found at minnpost.com/crime.

Updating data

Crime data comes from monthly reports produced by the Minneapolis PD. Due to the files being removed once, we upload the files to S3 so that we have our copy of the original data if needed. We then use ScraperWiki to store the data for us.

1. Download newest Excel file from the MPD site.
2. Rename to YYYY-MM.xlsx, for example 2014-06.xlsx.
3. Inspect Excel file and make sure the following.
   • The file is an .xlsx or .xls file. Both can be handled, but the former needs to be noted in the Scraper entry.
   • The first tab is the month. There will probably be other tabs for year-to-date, or quarterly data.
   • There are 10 columns, one for the neighborhood and the rest for stats.
   • There is only one line for the headers. This can be handled in the scraper if needed as well.
   • There are no empty lines before the last line. This can be handled in the scraper if needed as well.
4. Upload to AWS S3 in the following location: data.minnpost/projects/minnpost-crime/data-source/mpls-monthly-reports
   1. Make the item Public. This can be done by right-clicking the item and selecting Make Public.
   2. Use the Properties tab in the S3 interface to copy the URL to the new item.
5. Update the crime data scraper on ScraperWiki.
   1. Towards the end of the code, add an entry to the configuration dictionary. It should look like the most recent one. If some of the aspects (see above) of the file are not usual, you can provide some config for that; check out some of the older files.
      • year
      • month
      • url: This is the URL to the file on S3
      • original: This is not used at all and just for reference, but should be the URL to the file on the MPD site.
   2. Double check your code.
   3. Run the scraper. It should only run the most recent entry that you added; if you run again, it will end up processing all the files and this will take some time.
   4. Check the output and make sure there are no errors (there may be a warning about HTTPS, but this seems to be fine)
6. Wait an hour or (optional) Clear the data proxy. There is a proxy on Heroku that caches some of the ScraperWiki data and protects against ScraperWiki going down (which only happened once).
   1. Make sure that you have access to the MinnPost Heroku account and have Heroku setup on your computer.
   2. Get the proxy code from Github
   3. Connect code to Heroku: heroku git:remote -a all-good-proxy
   4. Restart app: heroku restart
7. Check the dashboard

Development and Install

Perquisites

1. Install Git. On Mac with Homebrew: brew install git
2. Install NodeJS. On Mac with Homebrew: brew install node
3. Install Grunt: npm install -g grunt-cli
4. Install Bower: npm install -g bower
5. Install Sass: gem install sass
6. Because Leaflet comes unbuilt, we need to build it with Jake: npm install -g jake

Get the code

1. Clone the repository with something like: git clone git@github.com:MinnPost/minnpost-crime.git
2. Go into the code directory: cd minnpost-crime

Install libraries

1. npm install
2. bower install
3. Because Leaflet comes unbuilt, we need to build it: cd bower_components/leaflet/ && npm install && jake; cd -;

Run

1. Us grunt to watch files for changes and serve the files with: grunt server-watch
2. Go to localhost:8899/index.html in your browser.

Build

1. grunt

Deploy

1. grunt mp-deploy

Data processing

1. To turn MN Compass neighborhood profile data to JSON: node data-processing/mncompass-2010-xlsx-json.js
2. Create final Minneapolis neighborhood data JSON:
   1. Run: node data-processing/minneapolis-neighborhoods.js

Data

Minneapolis

Crime

• Scraper of Minneapolis Monthly Reports. Original reports can be found on the MPD stats page. These reports are very similar to the FBI Uniform Crime Reports, except that they use the date of the offense (not reported date) and count each offense of the report (not the worse offense) see details.
  • (not applicable anymore) Due to the fact that the MPD has decided not to release Excel files anymore. This means that PDF's are manually parsed with Tabula and put into the data/crime/mpls-monthly-reports-manual-folder. These are then uploaded to S3 and used in the scraper.
  • Run node data-processing/download-mpls-monthly-reports.js to download the existing Excel files locally. Then do grunt mp-source-data to upload these files to S3.

Demographics

• MN Compass has a zipped Excel file of demographic data for Twin Cities neighborhoods that is downloaded here for processing. This is mostly 2010 population data with 2005-2009 employment and related data.
• The City of Minneapolis has a 2000 race and ethnicity report. A scraper of this data has been made. Download with: wget -O data/demographics/2000/minneapolis-demographics-2000.json "https://api.scraperwiki.com/api/1.0/datastore/sqlite?format=json&name=minneapolis_neighborhood_census_population_2000&query=select%20*%20from%20%60swdata%60"
  • Note that in 2005, the Phillips neighborhood was split up into Ventura Village, Phillips West, Easy Phillips, and Midtown Phillips (see neighborhood profile). The 2000 population is taken from Wikipedia:
    • http://en.wikipedia.org/wiki/Phillips_West,_Minneapolis
    • http://en.wikipedia.org/wiki/Phillips_East,_Minneapolis
    • http://en.wikipedia.org/wiki/Ventura_Village,_Minneapolis
    • http://en.wikipedia.org/wiki/Midtown_Phillips,_Minneapolis

Neighborhoods

• Definitive list of Minneapolis Neighborhoods last updated 2006-01-01. A scraper of this data has been made. Downloaded locally (data/neighborhoods/minneapolis/minneapolis-neighborhoods-2012.json). These contain keys that should be used throughout the application.
  • Download JSON with this command, though this data should not change anytime soon: wget -O data/neighborhoods/minneapolis/minneapolis-neighborhoods-2012.json "https://api.scraperwiki.com/api/1.0/datastore/sqlite?format=json&name=minneapolis_neighborhoods&query=select%20*%20from%20%60swdata%60"
  • Download CSV: wget -O data/neighborhoods/minneapolis/minneapolis-neighborhoods-2012.csv "https://api.scraperwiki.com/api/1.0/datastore/sqlite?format=csv&name=minneapolis_neighborhoods&query=select%20*%20from%20%60swdata%60"
• Downloaded shapefile of the neighborhoods was provided by the City of Minneapolis: wget -O data/neighborhoods/minneapolis/minneapolis-neighborhoods.shp.zip "http://www.minneapolismn.gov/www/groups/public/@bis/documents/webcontent/wcms1p-106980.zip". This was extracted with: cd data/neighborhoods/minneapolis/ && unzip minneapolis-neighborhoods.shpfile.zip -d minneapolis-neighborhoods.shpfile; cd -;
  • This has been converted to a GeojSON file with: ogr2ogr -f "GeoJSON" data/neighborhoods/minneapolis/minneapolis-neighborhoods.geo.json data/neighborhoods/minneapolis/minneapolis-neighborhoods.shpfile/NEIGHBORHOODS.shp -t_srs "EPSG:4326"

Hacks

• Currently using custom version of Backbone.stickit, see pull request.