Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
bin
 
 
 
 
lib
 
 
 
 
www
 
 
 
 
 
 
 
 
 
 
 
 

README.md

ottwatch

Code for ottwatch.ca - parts of which should be compatible with any SIREPUB municipal records management system

Installation

You will need PHP and MySQL 5.6 (beta right now). The newest MySQL is needed for the new GIS features that finally put it on par with PostGRES. I use PHP 5.3.10 and MySQL 5.6.10

The contents of /www is the root of the website. Everything else is used for background tasks and cron jobs.

Configuration is provided in lib/config.php. A sample file is provided in lib/config-sample.php

Scraping is performed by files in /bin. See the "Crontab" section below for ottwatch.ca's setup.

Database

  1. After creating a MySQL database the schema can be created via:

% mysql < lib/schema.sql

Example on Windows:

connect to your mysql databases and if your database name is ottwatchdb do

mysql> use ottwatchdb; 

mysql> source lib/opendata_parkland.sql;
  1. I've also provided SQL files for the opendata used so you don't have to worry about using "ogr2ogr" to convert from SHP files to Mysql GIS objects. You can load each of those as well with:

% for i in lib/opendata_*sql; do echo mysql < $i; done

Example of bash shell:

for OUTPUT in $(Linux-Or-Unix-Command-Here)

do 
	
	command1 on $OUTPUT
	
	command2 on $OUTPUT
	
	commandN on $OUTPUT

done

Crontab

Here is the crontab I use on ottwatch.ca to scrape data and also post it to Twitter

Scape the SIREpub RSS feed every 30 minutes looking for new meeting details

00,30 * * * * php /mnt/www/ottwatch/bin/meeting-parser.php

Once a day hard-scrape upcoming meetings even if the RSS doesn't indicate changes

00 5 * * * php /mnt/www/ottwatch/bin/meeting-parser.php hardScan

Tweet meeting updates during daytimes only

05,35 8-17 * * * php /mnt/www/ottwatch/bin/meeting-tweeter.php

Scape and tweet about development applications, daytime only

15,45 8-16 * * * php /mnt/www/ottwatch/bin/devapp-tweeter.php

Scrape the lobbyist registry, going back 30 days

20,50 * * * * php /mnt/www/ottwatch/bin/lobby-scaper.php 30

Tweet the lobbyist registry, daytime only

25,55 7-18 * * * php /mnt/www/ottwatch/bin/lobby-tweeter2.php

# Current crontab as of Jan-21-2014
*/5 * * * * php /mnt/www/ottwatch/bin/media-rss.php
*/5 * * * * php /mnt/www/ottwatch/bin/syndicate.php
00,30 * * * * php /mnt/www/ottwatch/bin/meeting-parser.php
05,35 * * * * php /mnt/www/ottwatch/bin/meeting-tweeter.php
00 5,15,19,22 * * * php /mnt/www/ottwatch/bin/meeting-parser.php hardScan
00 8-18 * * * php /mnt/www/ottwatch/bin/meeting-parser.php getVideos
00 8-18/2 * * * php /mnt/www/ottwatch/bin/meeting-parser.php getVideoStart
00 7,19 * * * php /mnt/www/ottwatch/bin/opendata.php 
15,45 * * * * php /mnt/www/ottwatch/bin/devapp-tweeter.php
20,50 * * * * php /mnt/www/ottwatch/bin/lobby-scaper.php 30
25,55 * * * * php /mnt/www/ottwatch/bin/lobby-tweeter2.php
00 8 * * * php /mnt/www/ottwatch/bin/election-countdown.php
0 7-21/2 * * * php /mnt/www/ottwatch/bin/consultations-parser.php
0 * * * * php /mnt/www/ottwatch/bin/public-notice-tweeter.php 
*/5 * * * * php /mnt/www/ottwatch/bin/page-monitor.php http://ottawa.ca/en/city-hall/your-city-government/elections/councillor
*/5 * * * * php /mnt/www/ottwatch/bin/page-monitor.php http://ottawa.ca/en/city-hall/your-city-government/elections/mayor
*/5 * * * * php /mnt/www/ottwatch/bin/disqus.php

Contributions

Many parts of this code are Ottawa specific, but the "Meeting" portions should be adaptable for any city that uses the SIREpub system. Good luck! I'm happy to merge "pull requests" if you send them. But really, probably the best path forward for other cities is to fork OttWatch into a generic SIREpub application and make that one better (and city generic).

License

About Me

You can find me on twitter at @odonnell_k

About

Code for ottwatch.ca - parts of which should be compatible with any SIREPUB municipal records management system

Resources

Releases

No releases published

Packages

No packages published
You can’t perform that action at this time.