In our democracy the definitive exercise of the power we give our politicians when we vote them into office is how they vote in our parliaments on our behalf. Yet you probably don't know how your MP votes. This isn't your fault.
Parliamentary voting information is notoriously difficult to find and analyse. This project changes that by making it understandable and easy to use.
Over 10 years ago the pioneering Public Whip project was created in the UK. This is an evolution of that original PHP application into a modern Rails application.
The OpenAustralia.org project parses the Australian Federal Hansard into ParlParse format (this due to it's history of being a fork of the UK TheyWorkForYou project). The debates XML files the parser creates, also available on data.openaustralia.org, contain voting data and we load this into a Rails application.
The UK Public Whip site still operates from the original PHP codebase however it's very possible for it to be upgraded to Rails in the future. During the development of They Vote For You, the OpenAustralia Foundation was careful to ensure there is an upgrade path.
To upgrade, checkout the
php-compatibility branch and point the Rails application at a copy of the UK production database. Test the site out and fix any bugs - there are likely to be some UK-specific additions needed to the Rails application.
Once the site is working you can then checkout a more recent version of the codebase and run
rake db:migrate to upgrade the database schema. This also is likely to need some UK-specific changes.
The final step is to customise the site language and interface. The best way to achieve this would be to develop some sort of theming system.
bundle exec rake application:load:popolo[https://raw.githubusercontent.com/everypolitician/everypolitician-data/master/data/Ukraine/Verkhovna_Rada/ep-popolo-v1.0.json]
Once the people data has been loaded you can start loading votes. These are scraped by another morph.io scraper, that saves data in a flat format that can easily be converted to Popolo. The conversion is handled by a small proxy application and the results are imported using another Rake task, e.g.:
bundle exec rake application:load:popolo[https://arcane-mountain-8284.herokuapp.com/vote_events/2015-07-14]
If your machine is already set up to develop Rails applications with MySQL just carry out the following steps and you should be good to go. Developing with Vagrant is also possible (see below) but was mainly useful with the retired PHP application.
Before beginning, install MySQL, HTMLTidy and Ruby:
# OS X ... brew install tidy-html5 mysql rbenv ruby-build rbenv install $(cat .ruby-version) # ... or Linux (Debian) sudo apt-get install tidy mysql-server mysql-client libmysqlclient-dev # then follow: https://github.com/sstephenson/rbenv#basic-github-checkout to get rbenv and ruby-build
Steps required to configure, install and start the Rails application:
# Install bundle bundle install # Copy the default config files over. # (Edit config/database.yml and fill in your username, password and database settings.) bundle exec rake application:config:dev cp config/database.yml.example config/database.yml # Copy secrets config cp config/secrets.yml.example config/secrets.yml # Set up your database (including seed data) bundle exec rake db:setup # Run tests bundle exec rake # Start the server bundle exec rails server
Once you have vagrant and virtualbox installed and have cloned this
vagrant up. This will download the base virtualbox image
and set up the development environment, be prepared for a bit of a wait.
Run the tests from inside the VM like this:
bundle exec rake
Assuming they pass, you can start the rails server:
bundle exec rails server
Once it is up you can browse to http://localhost:3000
If vagrant reports that it can't mount the
/vagrant virtualbox shared folder,
it's becuase the VM has had it's kernel updated. Run
vagrant provision && vagrant reload and you should be back in business.
The original PHP app is also available at http://localhost:8080 but only if you're running an older branch (out of scope for this guide).
These rake tasks are the ones you're most likely to need to run. You
can run them as the
deploy user in
/srv/www/production/current, for instance:
deploy@ip-172-31-37-36:/srv/www/production/current$ RAILS_ENV=production bundle exec rake application:load:divisions[2018-10-18]
application:load:membersloads members, offices and electorates. You always need this to run the site. Strictly speaking it only needs to run when details need updating but can be run as often as you like as it only updates data.
to_dateis optional and if omitted, allows you to load a single date.
application:cachethis namespace contains cache updating tasks that are necessary for the site to run. They should be self-explanatory.
Daily updates are carried out by the
application:load:daily Rake task,
which is run daily at 09:15 by cron.
Countries that use Popolo, e.g. Ukraine, only need to know about the
application:load:popolo Rake task. It will load people or country data, depending on what it finds in the file.
We're currently using a very old version of ElasticSearch in production, version 1.4.0, so we can't use the latest versions of the searchkick gem either.
Add data to your index the first time with
bundle exec rake searchkick:reindex:all and
Searchkick should take care of updates from there.
The code is deployed using Capistrano. To deploy to production run:
bundle exec cap production deploy
You'll need a local copy of
config/newrelic.yml that includes your licence
key to be able to record deployments to New Relic.
Ukraine's server has its configuration management in another repository. Once you've run the server provisioning tasks you can follow the instructions below to deploy the application.
After provisioning your development server, set up and deploy using Mina:
bundle exec mina ukraine-dev setup bundle exec mina ukraine-dev deploy # Now you can load people data bundle exec mina ukraine-dev rake[application:load:popolo[https://raw.githubusercontent.com/everypolitician/everypolitician-data/master/data/Ukraine/Verkhovna_Rada/ep-popolo-v1.0.json]] # And some vote data bundle exec mina ukraine-dev rake[application:load:popolo[https://arcane-mountain-8284.herokuapp.com/vote_events/2015-07-14]] # Setup caches bundle exec mina ukraine-dev rake[application:cache:all_except_member_distances] # Then build the index so search works bundle exec mina ukraine-dev rake[searchkick:reindex:all]
To deploy to the production server, replace
ukraine-production in the above commands.
This project uses some icons from the noun project under under creative commons licenses: