Parliamentary Monitoring Group website
Parliamentary monitoring application for use by the Parliamentary Monitoring Group in Cape Town, South Africa. See: https://www.pmg.org.za.
What does this project do
Allow citizens and other interested parties to monitor what's going on in the South African parliament. With specific focus on tracking the progress of legislation as it moves through the various phases: from being introduced for the first time to finally being approved and signed into law.
The purpose of the project is to improve parliamentary oversight, make the parliamentary process more accessible and improve transparency surrounding the activities of parliament.
How it works
The project consists of the following major components:
- User-facing website, including free and paid-for content (built using Flask, Jinja2 templates, Bootstrap and jQuery)
- Database (PostgreSQL)
- Search engine (Elastic Search)
- Admin interface (Flask-Admin, integration with Mandrill for email notifications)
- API (Flask)
- When this web app connects to its own API, it always connects to 127.0.0.1:5000 and sends the Host header of the
SERVER_HOSTto avoid routing "dogfooding" traffic to the outside internet.
Making use of the API
All of the data that is displayed through the frontend website, is served through an API at https://api.pmg.org.za which is freely accessible. However, please note that access to some content on the frontend website is restricted, and the same restrictions apply for the API.
Contributing to the project
This project is open-source, and anyone is welcome to contribute. If you just want to make us aware of a bug / make a feature request, then please add a new GitHub Issue (if a similar one does not already exist).
NOTE: On 2015-07-05 we removed some very large files from the repo and its history, reducing the size of the repo from over 100MB to 30MB. This required re-writing the history of the repo. You must pull and rebase your changes.
If you want to contribute to the code, please fork the repository, make your changes, and create a pull request.
Build the necessary services:
Setup the database:
docker-compose run --rm web python setup_dev_database.py docker-compose run --rm web python app.py db stamp head docker-compose run --rm web python bin/search.py --reindex all
Add the following lines to your .hosts file:
127.0.0.1 pmg.test 127.0.0.1 api.pmg.test
Start the server:
You can login with:
user : admin password : admin
Each time you pull in changes that might contain database changes:
docker-compose run --rm web python app.py db migrate docker-compose run --rm web python app.py db upgrade
To delete the database for a completely fresh setup, run:
docker-compose down --volumes
To start the task scheduler, run:
docker-compose run --rm web python app.py start_scheduler
Developing email features
Run a local mock SMTP server on port 2525
Set the SMTP environment variables
docker-compose -f docker-compose.yml -f docker-compose-test.yml run --rm web nosetests tests
We use Black to format our code. You can install it using
pip install black and run it with:
black app.py bin config pmg tests
Deployment is to dokku, a Heroku-like environment. To deploy, simply push to the git remote:
git push dokku
Sensitive or environment-specific configuration variables are set as environment variables using
dokku config:set, the important ones are:
- SERVER_NAME - Flask uses this as the base hostname and port for the server - Flask Blueprint subdomains base from this. If it can't match the Host header in requests to this, it serves a 404 response.
- Flask seems to use this for generating absolute URLs, except when the
X-Forwarded-Hostheader is provided, in which case that hostname is used for absolute URLs.
- FRONTEND_HOST - It's not currently clear if this is used anywhere
- STATIC_HOST=https://static.pmg.org.za/ or http://pmg-assets.s3-website-eu-west-1.amazonaws.com/
Reindexing for Search
To re-index all content for search, run:
ssh email@example.com run python bin/search.py --reindex all
This isn't normally necessary as the search index is updated as items are created, updated and deleted. It can be useful when the index has become out of date. Search functionality will fail while the indexing is in progress. Re-indexing takes about 10 minutes.
Dokku won't automatically start the task scheduler process. To start it, run:
dokku ps:scale pmg worker=1
python app.py db migrate -m "<revision description>"
Then to run the script on your local machine:
python app.py db upgrade
Updating parliamentary days
PMG needs to know the individual days in which Parliament sat, for each year. It uses this information
to calculate the number of parliamentary days that it took for bills to be adopted. It reads these days
from the file
Updating this information is a two-step process:
- Update the spreadsheet
data/parliament-sitting-days.xlsxthat lists the days parliament sits
python bin/load_parliamentary_days --pm-days data/parliament-sitting-days.xlsxto update
git diffto sanity check the changes
- Commit the changes
Application-level caching is used for certain views, initially based on which views the server spends most time on as seen in NewRelic Transaction overview.
To add caching to a view, add the following decorator - it must be the decorator closest to the view method so that it caches the view result, and not the result from other decorators:
from pmg import cache, cache_key, should_skip_cache ... @cache.memoize(make_name=lambda fname: cache_key(request), unless=lambda: should_skip_cache(request, current_user))
unlessmust be true when the cache should not be used. Frontend (views.py) views must always use this because the view shows them as logged in, even on pages where the rest of the data is the same. API views that don't serve subscription data or have any user-specific data don't need it.
make_namemust be the cache key for the view. It's very important that query strings are taken into consideration for the cache key.