Skip to content
No description or website provided.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bc211 Issue #502 bump version for bc211 data set Jun 7, 2019
common Issue #154 Renames from task to topic Jun 1, 2019
config Issue #364 Can retrieve related topics for a given service May 24, 2019
docker Issue #113 Rename docker files and directories Dec 13, 2017
features Issue #119 Create hack to make behave tests start with a clean databa… Nov 15, 2017
human_services Issue #154 Renames from task to topic Jun 1, 2019
main Issue #502 Update server version 1.3.1 Jun 7, 2019
newcomers_guide Issue #154 rename tasks arg to topics in import_newcomers_guide Jun 3, 2019
requirements Upgrade django from 2.2.1 to 2.2.2 Jun 14, 2019
search
taxonomies Issue #204 Move taxonomy module so that is can be easily accesses by … May 31, 2018
translation Issue #171 Rename translation Feb 7, 2018
users Issue #243 Add all Newcomers' Guilde languages except Tagalog Sep 20, 2018
utility Issue #154 Update SLQ script with some handy tricks for next time Jun 4, 2019
.coveragerc Issue #171 Rename translation Feb 7, 2018
.dockerignore
.editorconfig Issue #113 Clean up mess from cookiecutter in .gitignore and .editorc… Dec 19, 2017
.gitattributes Issue #113 Initialize environment from cookiecutter Dec 13, 2017
.gitignore Issue #317 fix intermittent test failure May 21, 2019
.pylintrc Issue #140 Clean up pylint warnings Jan 16, 2018
.travis.yml Issue #362 bump postgis to 2.4 May 21, 2019
LICENSE Create LICENSE Oct 20, 2017
Procfile Issue #170 Remove release tasks file until we can demonstrate a need … Mar 15, 2018
README.md Issue #154 Update README Jan 31, 2019
VERSION.txt Issue #502 Update server version 1.3.1 Jun 7, 2019
app.json Issue #154 Fix buildpack configuration Oct 18, 2018
compose-local.yml Issue #113 Rename docker files and directories Dec 13, 2017
compose-production.yml Issue #113 Rename docker files and directories Dec 13, 2017
env.example Issue #170 Update env.example to reflect new changes to config Mar 7, 2018
manage.py Issue #113 Move main/users app up to root directory Dec 19, 2017
pytest.ini Issue #113 Initialize environment from cookiecutter Dec 13, 2017
requirements.txt Issue #113 Copy in remaining cookiecutter files configuration files, … Dec 13, 2017
runtime.txt
setup.cfg Issue #113 Initialize environment from cookiecutter Dec 13, 2017

README.md

Pathways backend

This repository contains the server for providing access to data about services for refugees and immigrants to BC.

We can import and serve up read-only services data from the BC-211 dataset, using the HSDS data format and HSDA API format.

Getting started

Clone the repository

git clone git@github.com:pg-irc/pathways-backend.git

Set up and activate a python v3 environment

cd pathways-backend/
python3 -m venv .venv
source .venv/bin/activate

Install the required python libraries for local development, including the English SpaCy data set for natural language processing

pip install -r requirements/local.txt
python -m spacy download en

PostgreSQL on non-Ubuntu systems

Create a PostgreSQL user and database for your local development and enable the PostGIS extension. The user needs superuser permissions because the GIS extension needed for proximity search needs to be installed on test databases as part of the unit tests and only superusers are allowed to do that.

$ psql postgres
postgres=# CREATE USER pathways WITH PASSWORD 'your-secure-password';
postgres=# ALTER ROLE pathways SUPERUSER NOCREATEROLE CREATEDB;
postgres=# CREATE DATABASE pathways_local;
postgres=# GRANT ALL PRIVILEGES ON DATABASE pathways_local TO pathways;
postgres=# \connect pathways_local
postgres=# CREATE EXTENSION postgis;

Create a .env file and add your database user password

echo "POSTGRES_PASSWORD='your-secure-password'" > .env

To execute the test suite in production, where a database connection string is used, use a command line

DATABASE_URL=postgres://pathways:password@localhost/pathways_local \
MAILGUN_SENDER_DOMAIN=mailgun.org DJANGO_MAILGUN_API_KEY=xyz \
DJANGO_AWS_STORAGE_BUCKET_NAME=*bucket* \
DJANGO_SECRET_KEY=xyz ./manage.py test

PostgreSQL on Ubuntu systems

PostgreSQL is locked down and very secure by default on Ubuntu, so there is a simpler and more secure way to use on Ubuntu. Create a postgreql account with same name as your Linux account, ('rasmus' below) and make it superuser. You will be able to log into this postgresql account without a password:

$ sudo -u postgres createuser --interactive
Enter name of role to add: rasmus
Shall the new role be a superuser? (y/n) y

Log into postgresql as the root account (postgresql) and set up the database and set some properties on the user account:

$ sudo -u postgres psql
postgres=# ALTER ROLE rasmus SUPERUSER NOCREATEROLE CREATEDB;
postgres=# CREATE DATABASE pathways_local;
postgres=# GRANT ALL PRIVILEGES ON DATABASE pathways_local TO rasmus;

Now check that you can log in to psql as yourself with no password and that you have permission to install the postgis extension:

$ psql -d pathways_local
postgres=# CREATE EXTENSION postgis;

Now the .env file needs to contain the postgresql username only:

echo "POSTGRES_USER=rasmus" > .env

Create the database tables

python manage.py migrate

Create the django administration account:

python manage.py createsuperuser

Run the unit tests

python manage.py test

Start the API server

python manage.py runserver

You should now be able to access the server at http://127.0.0.1:8000/v1/. The Django admin tool is at http://127.0.0.1:8000/v1/admin/, and the question and choice entities are available at http://127.0.0.1:8000/v1/questions/ and http://127.0.0.1:8000/v1/questions/1/choices/.

Import BC-211 data

python manage.py import_bc211_data ~/path/to/AIRSXML_2252_Export_20170109050136__211.xml

Using different settings

By default the local settings are used. To use other settings, use the environment variable DJANGO_SETTINGS_MODULE, valid values are config.settings.local, config.settings.test and config.settings.production.

Running tests on Travis and locally

Travis runs the tests using the settings in config.settings.test, against postgres using the accont "postgres" with empty password, creating a database called "test_db". To run the same tests locally, create a postgres user and a database called "test_db" owned by that user, and specify the postgres account using environment variables POSTGRES_USER and POSTGRES_PASSWORD, either on the command line:

DJANGO_SETTINGS_MODULE=config.settings.test POSTGRES_USER=test_user POSTGRES_PASSWORD='the_password' python manage.py test

or put these settings in a file called .env (use env.example as a template) and export DJANGO_READ_DOT_ENV_FILE=True to make sure the file is read.

Upgrading packages

There is a utility script that may be used to update the python packages. Before using it, it may be a good idea to make sure that you have a correct environment based off of the requirements for the local environment.

deactivate
rm -r .venv-local/
python3 -m venv .venv-local
source ./.venv-local/bin/activate
pip install -r requirements/local.txt
python -m spacy download en

Then run the upgrade script for either local, rest or production environment, e.g.:

./utility/upgrade_packages.sh production

You will be prompted to decide which of the possible upgrades you will take (this is using pur in interactive mode), the requirement files are updated and a new environment is created using the new packages for use in testing before the upgrades are committed.

Deploying to Heroku

Prepare data

To deploy on Heroku from an empty production database: First prepare the deployment files locally using the utility/prepare_deploy.sh script. It should be called with three arguments

  • Path to the XML file containing the BC211 data
  • Path to the folder containing the Newcomers Guide content
  • Path to the output file to generate, must end in .json

Upload the resulting json file to AWS so that it can be accessed from Heroku.

Create the server

Ensure ALLOWED_HOSTS in the production settings include the name of the heroku instance.

These steps are for creating a new Heroku instance from scratch:

  • Create an instance on Heroku, give it a name
  • Under Resources, add a Heroku Postgres add-on, using the Hbby Basic level to support our amount of data
  • Under Deploy, connect it to github using the pg-irc organization and the pathways-backend repo
  • Under Settings, set all the environment variables (DATABASE_URL should already be there from the step above, you'll need that later)
Variable Value
DJANGO_SETTINGS_MODULE config.settings.production_heroku
DJANGO_READ_DOT_ENV_FILE False
BUILD_WITH_GEO_LIBRARIES 1
MAILGUN_SENDER_DOMAIN mailgun.org
DJANGO_AWS_STORAGE_BUCKET_NAME the bucket
DJANGO_SERVER_EMAIL the email
DJANGO_MAILGUN_API_KEY the key
DJANGO_SECRET_KEY the key
  • Under Deploy, Manual deploy, select git branch to deploy to the instance, then deploy it. This will take a while.

Populate the server

  • From bash, log into the server with
heroku ps:exec -a *appname*
  • Set environment variables in shell, using the DATABASE_URL from the Settings, see abovve
export DJANGO_SETTINGS_MODULE=config.settings.production
export DJANGO_READ_DOT_ENV_FILE=False
export MAILGUN_SENDER_DOMAIN="example.com"
export GDAL_LIBRARY_PATH="/app/.heroku/vendor/lib/libgdal.so"
export GEOS_LIBRARY_PATH="/app/.heroku/vendor/lib/libgeos_c.so"

export DJANGO_AWS_STORAGE_BUCKET_NAME="the bucket"
export DJANGO_SECRET_KEY="the key"
export DJANGO_MAILGUN_API_KEY="the key"
export DATABASE_URL="value from Settings"
  • Use wget to get the database dump file from AWS-S3
  • Load the data
./manage.py loaddata *file.json*
  • Then open the app, go to swagger and fire requests

Getting started with docker

Create and launch the docker containers for local development

docker-compose -f compose-local.yml build

Set up the database inside the container

docker-compose -f compose-local.yml run django python manage.py migrate
docker-compose -f compose-local.yml run django python manage.py createsuperuser

Launch the container

docker-compose -f compose-local.yml up

and check out http://localhost:8000/ to see if it worked. See https://cookiecutter-django.readthedocs.io/en/latest/developing-locally-docker.html for more details.

Getting started with PostgreSQL and PostGIS

PostgreSQL with the PostGIS extension is required for local development. You can find the installation that is right for your OS here: https://www.postgresql.org/download/ or use a package manager of your choice. The PostGIS extension can be found here: https://postgis.net/install/.

Translations

Certain content is translatable, using django-parler. The translation module provides management commands to export translatable strings as POT files, and import their translations from PO files.

List available translatable models:

./manage.py content_translation_list_models

Create a POT file for a particular translatable model:

./manage.py content_translation_export organizations.organization > organization.pot

Or create a PO file with a model's current French translations:

./manage.py content_translation_export organizations.organization --language=fr > fr/organization.po

The resulting files can be edited in an application such as GTranslator or Poedit.

After a translation file has been modified, you can import it again:

./manage.py content_translation_import fr/organization.po

Importing Newcomers Guide content

Converting the Newcomers' Giude content into a form that can be built into the client is done using a server side tool. This is an interrim measure only. Ultimately we will change this tool to instead import the same data into the database, hence it makese sense to build it on the server side.

Prepare the Newcomers Guide content as documented elsewhere. Convert the content from its plain Unicode text format to typescripts suitable for compiling into the client using the following server command:

./manage.py import_newcomers_guide path/to/newcomers/guide/content

This produces a number of typescript files in the working directory. Move these files into the client folder structure

mv *.ts ../pathways-frontend/src/fixtures/newcomers_guide/

Build the client as normal, noting that the screen output contains the line bin/buildFixtures.sh: Using Newcomers Guide fixtures to indicate that the Newcomers' Guide content is being used.

Development

Null and django blank

Required string fields should never be null and never be the empty string. Optional string fields should never be the empty string. To achieve this, blank parameters to field definitions should always be the same as the null parameter, e.g. either null=True, blank=True or null=False, blank=False, the latter being the default values for both fields, which can therefore be omitted.

Commit messages

All commits are labelled with the issue they are being done under. This ensures that we don't do work that is not tracked, and history of why every change is made is maintained. Most front end and back end work is tracked by issues in their respective repositories, in which case the commit message should start with "Issue #N", e.g. "Issue #13". Occasionally, front end work may be tracked under backend issues, in which case each commits message should start with "Issue pg-irc/pathways-backend#13".

You can’t perform that action at this time.