MadCrime is the Django app -- or rather Python scraper -- behind MadSafety, which is a table of Madison Police Department incident reports. From the list of reports, a user can click through to see the location of the incident and read the details.
Python
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
madcrime
.gitignore
README.md
requirements.txt

README.md

MadCrime:

A round-up of Madison Police Department Incident Reports

About


MadCrime is the Django app -- or rather Python scraper -- behind MadSafety, which is a table of Madison Police Department incident reports.

From the list of reports, a user can click through to see the location of the incident and read the details.

I hope to have MadFire -- a Python scraper for incident reports from the Madison Fire Department -- finished soon.

MadSafety

Links


Install & Run The Scraper


Download the zip file or fork the repo. Cd to your project directory and run pip install -r requirements.txt. I've been using the following for madcrime:

BeautifulSoup==3.2.1
Django==1.4
South==0.7.5
geopy==0.94.2
mechanize==0.2.5
mimeparse==0.1.3
python-dateutil==1.5
virtualenv==1.7.1.2
virtualenv-clone==0.2.4
virtualenvwrapper==3.5
wsgiref==0.1.2
yolk==0.4.3

Add the following to INSTALLED_APPS in settings.py:

'madcrime',

Add the following to urls.py:

(r'^incidents/', include('madcrime.urls')),

Run python manage.py syncdb from project directory.

To scrape run python manage.py scrapepd from your project directory. You will be asked which page you want to scrape. Enter 1 to the first page, etc.

  • Run python manage.py runserver and navigate to /incidents, and hopefully you see a list of incidents.

  • The scraper is located in madcrime/management/commands/scrapepd.py

Ideas to take this further


User Improvements

  • Add basic search capabilities.
  • Add pop up when user clicks on incident map.
  • Add map to main page with markers locating where incidents occured.
  • Table view should appear in descending order.
  • Sorting of incident table using tablesorter or datatables jQuery plugins.
  • Add date to map marker tooltip.
  • Add filter to main incident list so only incidents over last seven days appear.
  • Add incident reports from the Madison Fire Department.
  • Improve basic search to include plain text, search by date, search by incident type, search by address or search by radius.
  • Add map and pop ups to search results page.
  • Filter map markers on the incidents page by similar criteria.
  • Determine if content from Madison's Most Wanted and CrimeStoppers has a place.
  • Determine how to handle cases that aren't posted.
  • Add map with aldermanic ward polygons.
  • Alert system for when new incidents are loaded.
  • Hook in with Wis Circuit Court Access queries?
  • Search of incident table using datatables jQuery plugin.
  • Add layers of demographic data or other interesting tidbits.
  • Query madison.com and other local news sources for information about the incidents.

Admin Improvements

  • Sort incidents by release date in admin.
  • Rich text editor.
  • Ponder addition of a queue manager that holds incident details until they are edited/reviewed.
  • Template Improvements.

MVC Improvements

  • Find new method of geocoding addresses and storing them so Incidents model can access this information.
  • Change date of incident and date of incident report released to DateTime and Date fields.
  • Rename model information, or make json output dev friendly.
  • Consider the possibilities with django-bakery.

Scraper Improvements

  • Add command line input, asking user which page to scrape.
  • Use dateutil import parser to convert Date and Time information stored in incidentDetails into DateTime model.
  • Convert <br /> tags into periods.
  • Learn RegExp to add a space after periods found in suspect and arrested descriptions.
  • Added check against database for case number when scraping so to not lose edited addresses when running the scraper.
  • Cron job to run the scraper twice daily.
  • Learn RegExpressions and python methods to split suspect/arrest information -- stored in suspectDetails and arrestDetails -- into separate values.

License


The MIT License