Skip to content
Right to Education Index website
CSS Python HTML JavaScript Shell
Branch: master
Clone or download
roll Implement featured content (#34)
* Fix initial development steps

* Updated travis

* Generate an auto migration to get up-to-date

* Added blogpage.is_featured

* Added Featured Contentes block draft

* Implemented featured content gathering

* Limit and sort featured content

* Styled featured contents

* Update to Python@3.6.9
Latest commit 7c474d3 Oct 9, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.tx Setup Transifex, add docs Feb 18, 2016
bin Another go at upgrading setuptools Aug 30, 2016
data Data update after changes Jan 28, 2019
locale Out of School -> Out-of-school II May 10, 2017
media Fix exception on external URL documents Jan 16, 2019
rtei Implement featured content (#34) Oct 9, 2019
search Code changes for py2 to py3 transition Jan 14, 2019
.gitignore Implement featured content (#34) Oct 9, 2019
.travis.yml Implement featured content (#34) Oct 9, 2019
LICENCE Add AGPL 3 licence to repo Feb 3, 2016
Procfile Heroku ready Feb 10, 2016
README.md Implement featured content (#34) Oct 9, 2019
build_data.py Updates and fixes in data parsing Jan 28, 2019
data-requirements.txt
manage.py Initial commit Feb 1, 2016
requirements.in Upgrade requirements Jan 16, 2019
requirements.txt Upgrade requirements Jan 16, 2019
runtime.txt Implement featured content (#34) Oct 9, 2019
test-requirements.txt Pin django-webtest because of django-webtest/django-webtest#84 May 10, 2018

README.md

Right to Education Index (RTEI) website

Build Status

This site can be found at https://www.rtei.org and it is powered by Wagtail

Setup

  • Create a Postgres User and Database (password pass):

      sudo -u postgres createuser -d -S -R -P -l rtei
      sudo -u postgres createdb -O rtei rtei -E utf-8
    
  • Create a Virtualenv and clone this repo:

      mkvirtualenv rtei
      mkdir src && cd src
      git clone git@github.com:okfn/rtei.git
    
  • Install requirements:

      cd rtei
      pip install -r requirements.txt
    
  • Set up the database:

      python manage.py migrate
    
  • Create a super user and run the site:

      python manage.py createsuperuser
      python manage.py runserver
    

Custom Settings

RTEI requires the following values to be set as env vars:

SECRET_KEY='xxx'

# Email to receive contact requests from the form on /about/contact-us/
RTEI_CONTACT_FORM_EMAIL='contact_form@example.com'
EMAIL_HOST='xxx'
EMAIL_HOST_PASSWORD='xxx'
EMAIL_HOST_USER='xxx'

# S3 Storage for docs and resources
AWS_ACCESS_KEY_ID='xxx'
AWS_SECRET_ACCESS_KEY='xxx'
AWS_STORAGE_BUCKET_NAME='xxx'

# Postgres DB to use (dev defaults to postgres://rtei:pass@localhost/rtei)
DATABASE_URL='postgres:...'

# ElasticSearch endpoint use for search (dev defaults to http://localhost:9200)
ELASTICSEARCH_URL='http://example.com/es'

# Google Analytics site code for production
GOOGLE_ANALYTICS_CODE='xxx'

Notes for development

Tests

Install the tests requirements:

pip install -r test-requirements.txt

Run:

./manage.py test

Contents for development

Use the staging database's dump to populate contents for development. Copy images from the staging site to mimic the blog appearance.

Generate data

The JSON data needed to power the visualizations on the site is built using the build_data.py script, which parses the original Excel (xlsx) file located at data.

Before running the script you must install its requirements:

cd rtei
pip install -r data-requirements.txt

Run ./build_data.py -h to see all the options available.

Most of the times you will want to:

  1. Update /rtei/static/data/rtei_data_{year}.xlsx if necessary
  2. Run ./build_data.py all

The JSON data files are generated in rtei/static/data/{year} by default. These files are:

  • indicators.json: Master dictionary that links every indicator code to its title (and level). Indicators are nested, eg:

            [
                {
                    "code": "1",
                    "children": [
                        {
                            "code": "1.1",
                            "children": [
                                {
                                    "code": "1.1.1",
                                    "children": [
                                        {
                                            "code": "1.1.1a",
                                            "level": 4,
                                            "title": "The International Covenant of Economic, Social and Cultural Rights"
                                        },
                                        {
                                            "code": "1.1.1b",
                                            "level": 4,
                                            "title": "The Convention on the Rights of the Child"
                                        },
                                        ...
                                    ]
                                }
                        }
                }
                ...
            ]
  • themes.json: Master dictionary that links every theme code to its title (and level). Themes are nested, eg:

        [
            {
                "code": "1",
                "level": 1,
                "title": "Children with Disabilities"
                "children": [
                    {
                        "code": "1.A.A",
                        "level": 2,
                        "title": "Structure and Support"
                    }
                ]
            },
    
            ...
        ]
    
  • scores_per_country.json: Contains all level 1 and 2 scores for all countries, as well as the overall one:

            {
                "CL": {
                    "1": 64.76,
                    "1.1": 100
                    "1.2": 50
                    "1.3": 0
                    "1.4": 84.4,
                    "1.6": 89.4,
                    "2": 51.95,
                    "2.1": 33.3,
                    "2.2": 96
                    "2.3": 73.2,
                    "2.4": 5.3,
                    "3": 88.3833,
                    "3.1": 89.6,
                    "3.2": 83.25,
                    "3.3": 92.3,
                    "4": 80.8,
                    "4.1": 68.4,
                    "4.2": 77.7,
                    "4.3": 96.3,
                    "5": 55.4333,
                    "5.1": 66.5,
                    "5.2": 66.5,
                    "5.3": 33.3,
                    "index": 68.2653
                },
                "NG": {
                    "1": 94.56,
                    "1.1": 100
                    "1.2": 100
                    "1.3": 100
                    "1.4": 84.4,
                    "1.6": 88.4,
                    "2": 53.275,
                    "2.1": 25.6,
                    "2.2": 50
                    "2.3": 37.5,
                    "2.4": 100
                    "3": 77.8333,
                    "3.1": 97.4,
                    "3.2": 58.3,
                    "3.3": 77.8,
                    "4": 79.9667,
                    "4.1": 100
                    "4.2": 44.3,
                    "4.3": 95.6,
                    "5": 75.0667,
                    "5.1": 66.5,
                    "5.2": 92.7,
                    "5.3": 66
                    "index": 76.1403
                },
    
                ...
    
  • c3_scores_per_country.json: C3 optimized data for the country bar charts. Contains all level 1 and 2 scores normalized for all countries, as well as the overall one and the transversal themes:

        [
          {
            "1": 12.85,
            "1.1": 20
            "1.2": 10
            "1.3": 0
            "1.4": 16.88,
            "1.5": 17.39,
            "2": 15.13,
            "2.1": 8.32,
            "2.2": 24
            "2.3": 18.3,
            "2.4": 25
            "index": 72.
            "name": "Chile",
            "t1A.A": 68.4,
            "t2A": 1
            "t3A": 83.07,
            "t3B": 72.03,
            "t4A": 55.53,
            "t5A": 76.14,
            "t6A": 71.1,
            "t7A": 75,
            "t7B": 75.59,
            "t8A": 66.66,
            "t9A": 0
          },
          ...
    ]
  • {country_code}.json (eg CL.json): For each of the countries available, contains all values for all indicators for that particular country. The values are the user-friendly responses shown on the frontend:

            {
                "1": 64.267,
                "1.1": 100
                "1.1.1a": "Yes",
                "1.1.1b": "Yes",
                "1.1.1c": "Yes",
                "1.1.1d": "Yes",
                "1.1.1e": "Yes",
                "1.1.1f": "Yes",
                "1.1.1g": "Yes",
                "1.1.1h": "Yes",
                "1.1.2a": "Yes",
                "1.1.3a": "Yes",
                "1.1.3b": "Yes",
                "1.1.3c": "Yes",
                "1.1.4a": "Yes",
                "1.1.4b": "Yes",
                "1.1.4c": "Yes",
                "1.1.4d": "Yes",
                "1.1.5a": "Not Applicable",
                "1.1.5b": "Not Applicable",
                ...

Map data

The map visualizations are powered by a TopoJSON file. This is built using two input files, data/countries.geojson, which is constant and does not need to be updated, and data/scores_per_country.csv, which needs to be updated:

./build_data.py -o data scores-per-country-csv

Install the topojson command (use a version lower than 2.0):

npm install topojson@1.6.27

This is the full command that needs to be done (note that the output file goes to the rtei/static/data):

topojson -p -o rtei/static/data/countries.topojson --stitch-poles false --id-property iso2 -e data/scores_per_country.csv data/countries.geojson

Dump Site data

Dump a new set of site data (this is the internal data for managing the site, like pages etc, not the data powering the visualizations):

cd rtei
python manage.py dumpdata --natural-foreign --indent=4 -e contenttypes -e auth.Permission -e sessions -e wagtailcore.pagerevision -e auth.user > rtei/fixtures/data.json

Deployment

The site is hosted on Heroku:

  • rtei is the staging site, which is deployed automatically from Travis on each push to master.

  • rtei-production is the production site, deployed manually via git remote:

    heroku git:remote -r production -a rtei-production git push production master

Migrate data on Heroku

Database migrates can be run on heroku against the production settings with:

heroku run python ./manage.py migrate --settings=rtei.settings.production

Translations

Translations are managed on Transifex. You will need to install the Transifex command line client:

pip install transifex-client

If you haven't already done it, you need to create a ~/.transifexrc file with the following contents:

[https://www.transifex.com]
hostname = https://www.transifex.com
username = YOUR_USERNAME
password = YOUR_PASSWORD
token =

To test that it's properly configured, run the following on the repo directory:

tx status

You should see something along these lines:

rtei -> rtei (1 of 1)
Translation Files:
 - en: locale/django.pot (source)
 - ar: locale/ar/LC_MESSAGES/django.po
 - es: locale/es/LC_MESSAGES/django.po
 - fr: locale/fr/LC_MESSAGES/django.po

Uploading translations to Transifex

New strings added to the source code that need to be translated must be regularly extracted and uploaded to Transifex.

Note: This also includes strings in the original data! (eg Indicator titles or responses)

To do so run the following:

# Extract translatable strings from data
./build_data.py translation-strings

# Extract strings from source code into po files (also keep the master pot file)
./manage.py makemessages --keep-pot

# Upload to Transifex
tx push -s -t --skip

#Commit new po files
git commit -am "Update translation files with new strings"

At this point the strings are available for translation on Transifex.

Updating translations from Transifex

Once the translators have finished workin on Transifex, update the source code translations with the following commands:

# Pull strings from Transifex
tx pull

# Compile strings catalogue, ie the locale/*.mo files (You need to restart the server after this)
./manage compilemessages

# Commit the changes
git commit -am "Updated strings from Transifex"

Rebuilding CSS

The project uses SASS. To install it run:

sudo gem install sass

To rebuild the CSS, watching for changes in the scss files, run:

sass rtei/static/scss/rtei.scss:rtei/static/css/rtei.css --watch
You can’t perform that action at this time.