Code for ottwatch.ca - parts of which should be compatible with any SIREPUB municipal records management system
HTML Python PHP JavaScript OpenEdge ABL Java Other
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
config
lib
src/java/ca/ottwatch
vendor
www
.classpath
.gitignore
.project
Capfile
README.md
composer.json
composer.lock
composer.phar

README.md

ottwatch

Code for ottwatch.ca - parts of which should be compatible with any SIREPUB municipal records management system

Installation

You will need PHP and MySQL 5.6 (beta right now). The newest MySQL is needed for the new GIS features that finally put it on par with PostGRES. I use PHP 5.3.10 and MySQL 5.6.10

The contents of /www is the root of the website. Everything else is used for background tasks and cron jobs.

Configuration is provided in lib/config.php. A sample file is provided in lib/config-sample.php

Scraping is performed by files in /bin. See the "Crontab" section below for ottwatch.ca's setup.

Database

1) After creating a MySQL database the schema can be created via:

% mysql < lib/schema.sql

Example on Windows:

connect to your mysql databases and if your database name is ottwatchdb do

mysql> use ottwatchdb; 

mysql> source lib/opendata_parkland.sql;

2) I've also provided SQL files for the opendata used so you don't have to worry about using "ogr2ogr" to convert from SHP files to Mysql GIS objects. You can load each of those as well with:

% for i in lib/opendata_*sql; do echo mysql < $i; done

Example of bash shell:

for OUTPUT in $(Linux-Or-Unix-Command-Here)

do 

    command1 on $OUTPUT

    command2 on $OUTPUT

    commandN on $OUTPUT

done

Crontab

Here is the crontab I use on ottwatch.ca to scrape data and also post it to Twitter

# Scape the SIREpub RSS feed every 30 minutes looking for new meeting details

00,30 * * * * php /mnt/www/ottwatch/bin/meeting-parser.php

# Once a day hard-scrape upcoming meetings even if the RSS doesn't indicate changes

00 5 * * * php /mnt/www/ottwatch/bin/meeting-parser.php hardScan

# Tweet meeting updates during daytimes only

05,35 8-17 * * * php /mnt/www/ottwatch/bin/meeting-tweeter.php

# Scape and tweet about development applications, daytime only

15,45 8-16 * * * php /mnt/www/ottwatch/bin/devapp-tweeter.php

# Scrape the lobbyist registry, going back 30 days

20,50 * * * * php /mnt/www/ottwatch/bin/lobby-scaper.php 30

# Tweet the lobbyist registry, daytime only

25,55 7-18 * * * php /mnt/www/ottwatch/bin/lobby-tweeter2.php

# Current crontab as of Jan-21-2014
*/5 * * * * php /mnt/www/ottwatch/bin/media-rss.php
*/5 * * * * php /mnt/www/ottwatch/bin/syndicate.php
00,30 * * * * php /mnt/www/ottwatch/bin/meeting-parser.php
05,35 * * * * php /mnt/www/ottwatch/bin/meeting-tweeter.php
00 5,15,19,22 * * * php /mnt/www/ottwatch/bin/meeting-parser.php hardScan
00 8-18 * * * php /mnt/www/ottwatch/bin/meeting-parser.php getVideos
00 8-18/2 * * * php /mnt/www/ottwatch/bin/meeting-parser.php getVideoStart
00 7,19 * * * php /mnt/www/ottwatch/bin/opendata.php 
15,45 * * * * php /mnt/www/ottwatch/bin/devapp-tweeter.php
20,50 * * * * php /mnt/www/ottwatch/bin/lobby-scaper.php 30
25,55 * * * * php /mnt/www/ottwatch/bin/lobby-tweeter2.php
00 8 * * * php /mnt/www/ottwatch/bin/election-countdown.php
0 7-21/2 * * * php /mnt/www/ottwatch/bin/consultations-parser.php
0 * * * * php /mnt/www/ottwatch/bin/public-notice-tweeter.php 
*/5 * * * * php /mnt/www/ottwatch/bin/page-monitor.php http://ottawa.ca/en/city-hall/your-city-government/elections/councillor
*/5 * * * * php /mnt/www/ottwatch/bin/page-monitor.php http://ottawa.ca/en/city-hall/your-city-government/elections/mayor
*/5 * * * * php /mnt/www/ottwatch/bin/disqus.php

Contributions

Many parts of this code are Ottawa specific, but the "Meeting" portions should be adaptable for any city that uses the SIREpub system. Good luck! I'm happy to merge "pull requests" if you send them. But really, probably the best path forward for other cities is to fork OttWatch into a generic SIREpub application and make that one better (and city generic).

License

About Me

You can find me on twitter at @odonnell_k