Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
tree: 958d872b90
Fetching contributors…

Cannot retrieve contributors at this time

108 lines (87 sloc) 4.756 kb


A round-up of Madison Police Department Incident Reports


MadCrime is the Django app -- or rather Python scraper -- behind MadSafety, which is a table of Madison Police Department incident reports.

From the list of reports, a user can click through to see the location of the incident and read the details.

I hope to have MadFire -- a Python scraper for incident reports from the Madison Fire Department -- finished soon.


Install & Run The Scraper

Download the zip file or fork the repo. Cd to your project directory and run pip install -r requirements.txt. I've been using the following for madcrime:

django-haystack==2.0.0-beta (Needed to solve an issue I ran into. Your mileage may vary.)

Add the following to INSTALLED_APPS in


Add the following to

    'default': {
        'ENGINE': 'haystack.backends.simple_backend.SimpleEngine',

Add the following to

(r'^incidents/', include('madcrime.urls')),

Run python syncdb from project directory.

To scrape run python scrapepd from your project directory. You will be asked which page you want to scrape. Enter 1 to the first page, etc.

  • Run python runserver and navigate to /incidents, and hopefully you see a list of incidents.

  • The scraper is located in madcrime/management/commands/

Ideas to take this further

User Improvements

  • Add basic search capabilities.
  • Add pop up when user clicks on incident map.
  • Add map to main page with markers locating where incidents occured.
  • Table view should appear in descending order.
  • Sorting of incident table using tablesorter or datatables jQuery plugins.
  • Add date to map marker tooltip.
  • Add filter to main incident list so only incidents over last seven days appear.
  • Improve basic search to include plain text, search by date, search by incident type, search by address or search by radius.
  • Add map and pop ups to search results page.
  • Filter map markers on the incidents page by similar criteria.
  • Add incident reports from the Madison Fire Department.
  • Determine if content from Madison's Most Wanted and CrimeStoppers has a place.
  • Determine how to handle cases that aren't posted.
  • Add map with aldermanic ward polygons.
  • Alert system for when new incidents are loaded.
  • Hook in with Wis Circuit Court Access queries?
  • Search of incident table using datatables jQuery plugin.
  • Add layers of demographic data or other interesting tidbits.
  • Query and other local news sources for information about the incidents.

Admin Improvements

  • Sort incidents by release date in admin.
  • Rich text editor.
  • Ponder addition of a queue manager that holds incident details until they are edited/reviewed.
  • Template Improvements.

MVC Improvements

  • Find new method of geocoding addresses and storing them so Incidents model can access this information.
  • Change date of incident and date of incident report released to DateTime and Date fields.
  • Rename model information, or make json output dev friendly.
  • Consider the possibilities with django-bakery.

Scraper Improvements

  • Add command line input, asking user which page to scrape.
  • Use dateutil import parser to convert Date and Time information stored in incidentDetails into DateTime model.
  • Convert <br /> tags into periods.
  • Learn RegExp to add a space after periods found in suspect and arrested descriptions.
  • Added check against database for case number when scraping so to not lose edited addresses when running the scraper.
  • Learn RegExpressions and python methods to split suspect/arrest information -- stored in suspectDetails and arrestDetails -- into separate values.
  • Cron job to run the scraper twice daily.


The MIT License

Jump to Line
Something went wrong with that request. Please try again.