CountCulture / twfy_local_parser

Local scraper parser to scrape and parse local authority data

This URL has Read+Write access

Added specific location for filestore cache
CountCulture (author)
Tue Jun 30 05:36:22 -0700 2009
commit  a74b95e1e61b629ab2bdac2952b5740f44487d84
tree    dd0e4ae1db85d61ec03c750b6494b1208583fbc8
parent  cdc594483709fe0d6b38f42c6e0b95f004553a1e
name age
history
message
file .gitignore Wed Jun 24 01:31:31 -0700 2009 Added smtp_gmail gem to allow email to be sent ... [CountCulture]
file README Tue Jun 16 07:33:43 -0700 2009 Added share_block [CountCulture]
file Rakefile Sat Mar 07 01:50:39 -0800 2009 Initial commit. Plus added basic member model/c... [Chris Taggart]
directory app/ Loading commit data...
directory config/ Tue Jun 30 05:36:22 -0700 2009 Added specific location for filestore cache [CountCulture]
directory db/ Tue Jun 30 02:24:29 -0700 2009 Added Dataset#last_checked to separate stale fl... [CountCulture]
directory doc/ Sat Mar 07 01:50:39 -0800 2009 Initial commit. Plus added basic member model/c... [Chris Taggart]
directory lib/ Thu Jun 25 15:26:54 -0700 2009 Moved marking of council as updated when associ... [CountCulture]
directory log/ Sat Mar 07 01:50:39 -0800 2009 Initial commit. Plus added basic member model/c... [Chris Taggart]
directory public/
directory script/ Sat Mar 07 01:50:39 -0800 2009 Initial commit. Plus added basic member model/c... [Chris Taggart]
directory test/ Tue Jun 30 04:25:04 -0700 2009 View/CSS tweaks [CountCulture]
directory tmp/ Sat Mar 07 01:50:39 -0800 2009 Initial commit. Plus added basic member model/c... [Chris Taggart]
directory vendor/ Wed Jun 24 01:31:31 -0700 2009 Added smtp_gmail gem to allow email to be sent ... [CountCulture]
README
== twfy_local_parser is a scraper/parser for local authorities.

A live version can be seen at http://theyworkforyoulocal.com

At the moment it’s in the early stages, concentrating on developing a scraper/parser system where the scraper/parsers ar
e stored in the db (via a web interface).

Anyone wishing to help on this project, please email me at countculture at googlemail dot com

Repository and bug tracker at http://github.com/CountCulture/twfy_local_parser

Copyright (c) 2009 Chris Taggart.
countculture at googlemail dot com;  http://countculture.wordpress.com/


BUGS
* When adding join model (e.g. committee memberships), currently added even when test processing. Plus no reporting, as 
reporting uses Dirty, which doesn't act on associations
* When doing test scrape of member info new info (eg email addresses) shows up, but doesn't when processing