A round-up of Madison Police Department Incident Reports
From the list of reports, a user can click through to see the location of the incident and read the details.
I hope to have MadFire -- a Python scraper for incident reports from the Madison Fire Department -- finished soon.
Install & Run The Scraper
Download the zip file or fork the repo. Cd to your project directory and run pip install -r requirements.txt. I've been using the following for madcrime:
BeautifulSoup==3.2.1 Django==1.4 South==0.7.5 geopy==0.94.2 mechanize==0.2.5 mimeparse==0.1.3 python-dateutil==1.5 virtualenv==22.214.171.124 virtualenv-clone==0.2.4 virtualenvwrapper==3.5 wsgiref==0.1.2 yolk==0.4.3
Add the following to INSTALLED_APPS in settings.py:
Add the following to urls.py:
Run python manage.py syncdb from project directory.
To scrape run python manage.py scrapepd from your project directory. You will be asked which page you want to scrape. Enter 1 to the first page, etc.
Run python manage.py runserver and navigate to /incidents, and hopefully you see a list of incidents.
The scraper is located in madcrime/management/commands/scrapepd.py
Ideas to take this further
Add basic search capabilities. Add pop up when user clicks on incident map. Add map to main page with markers locating where incidents occured. Table view should appear in descending order. Sorting of incident table using tablesorter or datatables jQuery plugins. Add date to map marker tooltip. Add filter to main incident list so only incidents over last seven days appear. Add incident reports from the Madison Fire Department.
- Improve basic search to include plain text, search by date, search by incident type, search by address or search by radius.
- Add map and pop ups to search results page.
- Filter map markers on the incidents page by similar criteria.
- Determine if content from Madison's Most Wanted and CrimeStoppers has a place.
- Determine how to handle cases that aren't posted.
- Add map with aldermanic ward polygons.
- Alert system for when new incidents are loaded.
- Hook in with Wis Circuit Court Access queries?
- Search of incident table using datatables jQuery plugin.
- Add layers of demographic data or other interesting tidbits.
- Query madison.com and other local news sources for information about the incidents.
Sort incidents by release date in admin.
- Rich text editor.
- Ponder addition of a queue manager that holds incident details until they are edited/reviewed.
- Template Improvements.
Find new method of geocoding addresses and storing them so Incidents model can access this information. Change date of incident and date of incident report released to DateTime and Date fields.
- Rename model information, or make json output dev friendly.
- Consider the possibilities with django-bakery.
Add command line input, asking user which page to scrape. Use dateutil import parser to convert Date and Time information stored in incidentDetails into DateTime model. Convert <br /> tags into periods. Learn RegExp to add a space after periods found in suspect and arrested descriptions. Added check against database for case number when scraping so to not lose edited addresses when running the scraper. Cron job to run the scraper twice daily.
- Learn RegExpressions and python methods to split suspect/arrest information -- stored in suspectDetails and arrestDetails -- into separate values.
The MIT License