chivote

• Requirements
• Installation
• Env variables
• Management commands
• Internationalization
• Archiving
• Production use
• Under the hood
• References
• Todos

Requirements

Local dependency	Mac setup	Ubuntu setup
python 3.6	🔗	🔗
PostgreSQL	🔗	🔗
pipenv	🔗	🔗
node	🔗	🔗
yarn	🔗	🔗

🔝

Installation

1. Clone repo
2. Set up postgres database

sudo su - postgres
psql
CREATE USER sample_user WITH PASSWORD 'sample_password';
CREATE DATABASE sample_database WITH OWNER sample_user;
ALTER USER sample_user WITH SUPERUSER

3. From inside the repo, create .env and add postgres credentials

touch .env
echo "PG_NAME = sample_database
PG_USER = sample_user
PG_PASSWORD = sample_password" >> .env

4. Install python requirements and activate virtualenv with pipenv sync && pipenv shell
5. Load initial database:

./manage.py migrate
./manage.py createsuperuser
./manage.py collectstatic

6. Build frontend with yarn --cwd ./frontend install

~ ~ ~

If you want to match your local db to the production db, here's an alias command to drop into ~/.bash_profile or a similar terminal management file.

alias cvdb='pg_dump chivote > /tmp/chivote.bk.psql; dropdb chivote; ssh -i /path/to/chivote.pem ubuntu@ec2-54-236-199-60.compute-1.amazonaws.com pg_dump chivote > /tmp/chivote.psql; createdb chivote; psql chivote < /tmp/chivote.psql'

🔝

Env variables

In production, you need a .env file in the root directory with the following filled out:

DJANGO_SECRET_KEY=
DJANGO_DEBUG=False
DJANGO_URL_ENDPOINT=

PG_NAME=
PG_USER=
PG_PASSWORD=

## datadesk/django-bakery settings ##
AWS_BUCKET_NAME=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=
ALLOW_BAKERY_AUTO_PUBLISHING=
BAKERY_GZIP=True

BALLOT_READY_API_KEY=
BALLOT_READY_API_URL=

CELERY_BROKER_URL=
IL_SUNSHINE_API_URL=

CHIVOTE_IS_RUNOFF=
# CHIVOTE_URL_PREFIX=
# CHIVOTE_ARCHIVE_MESSAGE=

🔝

Management commands

Launch dev environment: ./manage.py serve

Launches dev environment at http://localhost:8000/. It simply starts the various servers in one terminal instead of three.

# this is pseudo python describing the task

devCommands = [
    # redis server
    'redis-server',

    # celery worker
    'celery -A chivote worker -l info',

    # frontend webpack-dev-server w/ hot module replacement
    'yarn --cwd ./frontend start',

    # django-livereload-server, to automatically reload browser on a Django file change
    'python manage.py livereload --settings=chivote.settings.local',

    # django server
    'python manage.py runserver --settings=chivote.settings.local',
]

for command in devCommands:
    # do command
    ...

Launch production environment: ./manage.py serve --production

Launches production environment at http://localhost:8000/. It simply runs the necessary builders to bake the app out as flat files, then serves those files.

# this is pseudo python describing the task

prodCommands = [
    # frontend production build
    'yarn --cwd ./frontend build',

    # django collectstatic (incl. built frontend)
    'python manage.py collectstatic --no-input --settings=chivote.settings.production',

    # django-bakery build
    'python manage.py build --settings=chivote.settings.production',

    # django-bakery production server
    'python manage.py buildserver --settings=chivote.settings.production',
]

for command in prodCommands:
    # do command
    ...

🔝

Internationalization

TODO: explain django side and frontend side

From inside frontend, run yarn build:langs to generate public/locales/data.json. This compiles public/locales/messages/* into a single message file, as well as any locale files that are in public/locales.

TODO: Automate locale file generation (e.g. public/locales/es.json).

TODO: Automate syncing local json to Google Sheet.

TODO: Add compilemessages, etc. to rebuild.

🔝

Archiving

As is, our site can't handle multiple elections. To accomplish that, we'd need to create an election model, create race instances per elections, create candidate instances per race instance per election... So instead, we're doing this in a dumb, destructive way.

• Generate a prefixed build of the site by setting CHIVOTE_URL_PREFIX

# add to .env

CHIVOTE_URL_PREFIX = 'archive/2019-feb-26/'
CHIVOTE_ARCHIVE_MESSAGE='Archived: March 15, 2019'

• That prefixed build should host static versions of BallotReady data and results
• Upload that prefixed build to the s3 bucket
• From AWS, manually protect those archive folders from being deleted
• Tag the version in git
• Upload a copy of the db to s3 bucket/private/pg_dump*.psql

At this point, our archived version should be treated like it's dead and buried. Time to move on.

This 👏 is 👏 not 👏 great. 👏 But it's what we've got.

🔝

SSR

see frontend/package.json for render-server commands

TK

🔝

Production use

Server

Our app is deployed on an EC2 instance. I used these instructions from DigitalOcean for setting it up. For continuing maintenance and troubleshooting, read through those instructions' troubleshooting section .

Celery

Instructions for Celery setup and maintenance are documented in the pull request that first integrated Celery.

BallotReady data

BallotReady data is loaded live from their API via a proxy API we've set up in order to hide our BallotReady API key and monitor API use. Our API is hosted on AWS API Gateway.

Updating code

The public site is hosted on an S3 bucket. On the server, django-bakery and celery manage automatic data updates of the site. The one caveat here is if a data update happens while code is also being updated, there's a good chance that the S3 version will get all kinds of mucked up.

With that in mind, here's the process for updating code. Eventually this should be automated, but for now, you need to run each command in sequence:

sudo supervisorctl stop all # stops celery server from uploading to s3
git pull
./rebuild.sh
sudo supervisorctl start chivote_render
sudo systemctl restart gunicorn
pipenv run ./manage.py build --settings=chivote.settings.production
pipenv run ./manage.py publish
sudo supervisorctl start all # resume celery server uploads to s3

🔝

Under the hood

Baking

We're using datadesk/django-bakery to bake out our app as flat files and to publish those files to s3. See their docs for further instruction.

Connecting Django and React

Loading the frontend files into Django templates is accomplished with owais/django-webpack-loader.

In the frontend, Webpack creates a stats file called webpack-stats.json. Django uses that stats file to load webpack chunks in a template through the function render_bundle [chunk_name].

We pass data from Django to React by declaring global variables in our Django templates. I adapted this pattern from MasterKale/django-cra-helper.

Overview of flow:

• View: Exposes context data for Template
• Template: Exposes context data as JavaScript vars for Index, attaches frontend files from Webpack-Stats
• Webpack-Stats: References built output of compiled Index
• Index: Renders Component, w/ props
• Component: Loaded with props exposed in Template

Full example:

myapp/views.py (View)

from django.views import generic

class HomepageView(generic.TemplateView):
    """View function for home page of site."""
    template_name = 'index.html'

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)

        context_dict = {
            'env': 'django'
        }

        react_dict = {
            'component': 'App',
            'props': context_dict,
        }
        context.update(react_dict)

        return context

myapp/templates/index.html (Template)

{% load render_bundle from webpack_loader %}

<div id="app"></div>

<script>
  window.component = '{{ component }}';
  window.props = {{ props | json }};
  window.reactRoot = document.getElementById('app');
</script>

{% render_bundle 'main' %}

frontend/webpack-stats.json (Webpack-Stats)

// this all builds from webpack, with an entry of frontend/src/index.js
{
  "status":"done",
  "publicPath":"http://localhost:3000/static/dist/",
  "chunks":{
    "main":[{
      "name":"main-4a3ebe49ac244ea51884.js",
      "publicPath":"http://localhost:3000/static/dist/main-4a3ebe49ac244ea51884.js",
      "path":"/Users/pjudge/bettergov/projects/chivote/frontend/dist/main-4a3ebe49ac244ea51884.js"
    }]
  }
}

frontend/src/index.js (Index)

import React from 'react';
import ReactDOM from 'react-dom';
import App from './App';

/**
 * Maintain a simple map of React components to make it easier for
 * Django to reference individual components.
 */

const pages = {
  App
};

/**
 * If Django hasn't injected these properties into the HTML
 * template that's loading this script then we're viewing it
 * via the create-react-app liveserver
 */
window.component = window.component || 'App';
window.props = window.props || { env: 'create-react-app' };
window.reactRoot = window.reactRoot || document.getElementById('root');

ReactDOM.render(
  React.createElement(pages[window.component], window.props),
  window.reactRoot
);

frontend/src/App.jsx (Component)

import React, { Component } from 'react';

class App extends Component {
  render() {
    return (
      <div>
        <h1>Hello world!</h1>
        <h2>Env: {this.props.env}</h2>
      </div>
    );
  }
}

export default App;

🔝

References

This React-in-Django approach is informed by a few articles:

• "Reconciling Backend Templates with Frontend Components," Nick Sweeting
• "Using Webpack transparently with Django + hot reloading React components as a bonus," Owais Lone
• Politico Interactive News on public-facing apps

🔝

Todos

(clean for now)

🔝