Parliamentary Monitoring Group website
Parliamentary monitoring application for use by the Parliamentary Monitoring Group in Cape Town, South Africa. See: https://www.pmg.org.za.
What does this project do
Allow citizens and other interested parties to monitor what's going on in the South African parliament. With specific focus on tracking the progress of legislation as it moves through the various phases: from being introduced for the first time to finally being approved and signed into law.
The purpose of the project is to improve parliamentary oversight, make the parliamentary process more accessible and improve transparency surrounding the activities of parliament.
How it works
The project consists of the following major components:
- User-facing website, including free and paid-for content (built using Flask, Jinja2 templates, Bootstrap and jQuery)
- Database (PostgreSQL)
- Search engine (Elastic Search)
- Admin interface (Flask-Admin, integration with Mandrill for email notifications)
- API (Flask)
Making use of the API
All of the data that is displayed through the frontend website, is served through an API at https://api.pmg.org.za which is freely accessible. However, please note that access to some content on the frontend website is restricted, and the same restrictions apply for the API.
Contributing to the project
This project is open-source, and anyone is welcome to contribute. If you just want to make us aware of a bug / make a feature request, then please add a new GitHub Issue (if a similar one does not already exist).
NOTE: On 2015-07-05 we removed some very large files from the repo and its history, reducing the size of the repo from over 100MB to 30MB. This required re-writing the history of the repo. You must pull and rebase your changes.
If you want to contribute to the code, please fork the repository, make your changes, and create a pull request.
You'll also need the psql and libxml development libraries.
# Ubuntu/debian sudo apt-get install libpq-devel libxml2-dev libxslt1-dev python-dev # macOS brew install libxml2
You'll need python 2.7 and virtualenv.
Clone this repo, and setup a virtualenv:
virtualenv --no-site-packages env source env/bin/activate
pip install -r requirements.txt
Add the following lines to your
127.0.0.1 api.pmg.test 127.0.0.1 pmg.test
Create the pmg user with password
pmg, and an empty database:
createuser pmg -P createdb -O pmg pmg
Get a copy of the production database from a colleague, or setup a blank database. If you have a database copy, run:
gunzip -c pmg.sql.gz | psql -U pmg
Start the server:
python app.py runserver
Developing email features
Run a local mock SMTP server on port 2525
Set the SMTP environment variables
Create a test database:
psql -c 'create database pmg_test' psql -c 'grant all privileges on database pmg_test to pmg'
Then run the tests:
Deployment is to dokku, a Heroku-like environment. To deploy, simply push to the git remote:
git push dokku
Sensitive configuration variables are set as environment variables using Heroku or
dokku config:set, the important ones are:
Reindexing for Search
To re-index all content for search, run:
ssh email@example.com run python bin/search.py --reindex all
This isn't normally necessary as the search index is updated as items are created, updated and deleted. It can be useful when the index has become out of date. Search functionality will fail while the indexing is in progress. Re-indexing takes about 10 minutes.
python app.py db migrate -m "<revision description>"
Then to run the script on your local machine:
python app.py db upgrade
Updating parliamentary days
PMG needs to know the individual days in which Parliament sat, for each year. It uses this information
to calculate the number of parliamentary days that it took for bills to be adopted. It reads these days
from the file
Updating this information is a two-step process:
- Update the spreadsheet
data/parliament-sitting-days.xlsxthat lists the days parliament sits
python bin/load_parliamentary_days --pm-days data/parliament-sitting-days.xlsxto update
git diffto sanity check the changes
- Commit the changes
Application-level caching is used for certain views, initially based on which views the server spends most time on as seen in NewRelic Transaction overview.
To add caching to a view, add the following decorator - it must be the decorator closest to the view method so that it caches the view result, and not the result from other decorators:
from pmg import cache, cache_key, should_skip_cache ... @cache.memoize(make_name=lambda fname: cache_key(request), unless=lambda: should_skip_cache(request, current_user))
unlessmust be true when the cache should not be used. Frontend (views.py) views must always use this because the view shows them as logged in, even on pages where the rest of the data is the same. API views that don't serve subscription data or have any user-specific data don't need it.
make_namemust be the cache key for the view. It's very important that query strings are taken into consideration for the cache key.