This document is intended to guide you through setting up a local development environment for the Colouring Cities application. This guide assumes you already have either already have access to an machine with Ubuntu 18.04 installed, or can use VirtualBox to set up an Ubuntu virtual machine as below.
This guide assumes you are working with the 'colouring-core' repository. If you are creating your own fork, or want to use a custom city name, then you may wish to change 'colouring-core' to 'colouring-[your city name]'.
Configuring an Ubuntu VM in VirtualBox
Here we explain how to use VirtualBox and SSH into your Ubuntu installation for convenience.
When setting up the VirtualBox VM, consider the size of the database you intend to load for use with the application. Consult the 🏠 Loading the building data section of this guide and decide whether you will be using a full city database or will load test data from OSM.
For "Colouring London", we have found that the size of the database means that a VM with access to 50GB of storage is appropriate. If you are using the OSM test data, the default storage settings in VirtualBox should suffice.
If you a running Ubuntu in a virtual environment you will need to configure networking to forward ports from the guest to the host. For Virtual Box the following was configured under NAT port forwarding (found under Settings -> Network -> Advanced -> Port Forwarding).
| Name | Protocol | Host Port | Guest Port |
|---|---|---|---|
| app | TCP | 8080 | 3000 |
| app_dev | TCP | 3001 | 3001 |
| ssh | TCP | 4022 | 22 |
The app_dev mapping is used in development by Razzle which rebuilds and serves client side assets on the fly.
To run the commands in the rest of this setup guide, either ssh into the VirtualBox environment or open the terminal within the Ubuntu GUI.
If you wish to ssh, you will first need to open the terminal in Ubuntu and run the following.
sudo apt-get install -y openssh-serverYou can then ssh into the VirtualBox VM set up with the port forwarding described above like so, where <linuxusername> is the name you set up during the installation of Ubuntu (you can type whoami in the Ubuntu terminal to remind yourself of this).
ssh <linuxusername>@localhost -p 4022First upgrade the installed packages to the latest versions, to remove any security warnings.
sudo apt-get update -y
sudo apt-get upgrade -yNow install some essential tools.
sudo apt-get install -y build-essential git wget curl parallel renameSet the postgres repo for apt (these instructions were taken from postgresql.org).
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'sudo wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -sudo apt-get updateNext install postgres and postgis to enable support for geographical objects.
sudo apt-get install -y postgresql-12 postgresql-contrib-12 libpq-dev postgis postgresql-12-postgis-3and additional geo-spatial tools
sudo apt-get install -y gdal-bin libspatialindex-dev libgeos-dev libproj-devNow clone the colouring-core codebase.
cd ~ && git clone https://github.com/colouring-cities/colouring-core.gitNote: We assume here that you will clone the repo into the home directory of your Ubuntu installation. Watch out for later commands in this guide that assume the repo is located at ~/colouring-core and modify the path if appropriate.
Now install Node. It is helpful to define some local variables.
export NODE_VERSION=v16.13.2
export DISTRO=linux-x64
wget -nc https://nodejs.org/dist/$NODE_VERSION/node-$NODE_VERSION-$DISTRO.tar.xz
sudo mkdir /usr/local/lib/node
sudo tar xf node-$NODE_VERSION-$DISTRO.tar.xz -C /usr/local/lib/node
sudo mv /usr/local/lib/node/node-$NODE_VERSION-$DISTRO /usr/local/lib/node/node-$NODE_VERSION
rm node-$NODE_VERSION-$DISTRO.tar.xzNow add the Node installation to the path and export this to your bash profile.
cat >> ~/.profile <<EOF
export NODEJS_HOME=/usr/local/lib/node/node-$NODE_VERSION/bin
export PATH=\$NODEJS_HOME:\$PATH
EOFThen run source to make sure node and npm are on your path.
source ~/.profileYou can check the updated variables as follows
echo $PATH
echo $NODEJS_HOMENow we configure postgres. First ensure postgres is running.
sudo service postgresql startEnsure the en_US locale exists.
sudo locale-gen en_US.UTF-8Configure postgres to listen on network connection.
sudo sed -i "s/#\?listen_address.*/listen_addresses '*'/" /etc/postgresql/12/main/postgresql.confAllow authenticated connections from any IP (so includes the host).
echo "host all all all md5" | sudo tee --append /etc/postgresql/12/main/pg_hba.conf > /dev/nullRestart postgres to pick up config changes.
sudo service postgresql restartCreate a superuser role for this user (<username>) if it does not already exist. The
password <pgpassword> is arbitrary and probably should not be your Ubuntu login password.
sudo -u postgres psql -c "SELECT 1 FROM pg_user WHERE usename = '<username>';" | grep -q 1 || sudo -u postgres psql -c "CREATE ROLE <username> SUPERUSER LOGIN PASSWORD '<pgpassword>';"Note for "Colouring Cities" devs
If you intend to load the full CL database from a dump file into your dev environment, run the above psql command with <username> as "cldbadmin" and use that username in subsequent steps, but also run the above a second time with <username> as "clwebapp" (see section 🏠 Loading the building data for more details).
Now create an empty database configured with geo-spatial tools. The database name (<colouringcitiesdb>) is arbitrary.
Set environment variables, which will simplify running subsequent psql commands.
export PGPASSWORD=<pgpassword>
export PGUSER=<username>
export PGHOST=localhost
export PGDATABASE=<colouringcitiesdb>Create the database.
sudo -u postgres psql -c "SELECT 1 FROM pg_database WHERE datname = '<colouringcitiesdb>';" | grep -q 1 || sudo -u postgres createdb -E UTF8 -T template0 --locale=en_US.utf8 -O <username> <colouringcitiesdb>psql -c "create extension postgis;"
psql -c "create extension pgcrypto;"
psql -c "create extension pg_trgm;"Now upgrade the npm package manager to the most recent release with global privileges. This needs to be performed as root user, so it is necessary to export the node variables to the root user profile.
export NODEJS_HOME=/usr/local/lib/node/node-v16.13.2/bin/
export PATH=$NODEJS_HOME:$PATH
sudo env "PATH=$PATH" npm install -g npm@latestNow install the required Node packages. This needs to done from the app directory of your
local repository, so that it can read from the package.json file.
cd ~/colouring-core/app
npm installInstall python and related tools.
sudo apt-get install -y python3 python3-pip python3-dev python3-venvThere are several ways to create the Colouring Cities database in your environment. The simplest way if you are just trying out the application would be to use test data from OSM, but otherwise you should follow one of the instructions below to create the full database either from scratch, or from a previously made db (via a dump file).
To create the full database from scratch, follow these instructions, otherwise choose one of the following:
Create database from dump
If you are a developer on the Colouring London project (or another Colouring Cities project), you may have a production database (or staging etc) that you wish to duplicate in your development environment.
Log into the environment where your production database is kept and create a dump file from the db.
pg_dump <colouringcitiesdb> > <dumpfile>You should then download the file to the machine where you are setting up your development environment. If you are using Virtualbox, you could host share the dump file with the VM via a shared folder (e.g. see these instructions for Mac).
In your Ubuntu installation where you have been running these setup steps (e.g. Virtualbox VM), you can then recrate the db like so.
psql < <dumpfile>Alternatively, if you get errors using the above command, use pg_restore:
pg_restore -d <colouringcitiesdb> <dumpfile>Now run all 'up' migrations to create tables, data types, indexes etc. The .sql scripts to
do this are located in the migrations folder of your local repository.
ls ~/colouring-core/migrations/*.up.sql 2>/dev/null | while read -r migration; do psql < $migration; done;Again, if you get errors, you may need to manually specify the database name
ls ~/colouring-core/migrations/*.up.sql 2>/dev/null | while read -r migration; do psql -d <colouringcitiesdb> < $migration; done;Create database with test data
This section shows how to load test buildings into the application from OpenStreetMaps (OSM).
Create a virtual environment for python in the etl folder of your repository. In the following example we have name the virtual environment colouringcities but it can have any name.
cd ~/colouring-core/etl
pyvenv colouringcitiesActivate the virtual environment so we can install python packages into it.
source colouringcities/bin/activateInstall python pip package manager and related tools.
pip install --upgrade pip
pip install --upgrade setuptools wheelInstall the required python packages.
pip install -r requirements.txtTo help test the Colouring Cities application, get_test_polygons.py will attempt to save a small (1.5km²) extract from OpenStreetMap to a format suitable for loading to the database.
NOTE: You can edit the file by changing this line to specify the latitude and longitude used to define the centre of the sample data area, as well as the distance from that point, which are used to generate the sample data. (i.e. you could change it to match the cc-config.json file - as discussed in configuring-colouring-cities.md.
Download the test data.
python get_test_polygons.pyNote: the first time you run it, you will get these warnings:
rm: cannot remove 'test_buildings.geojson': No such file or directory
rm: cannot remove 'test_buildings.3857.csv': No such file or directory
Now run all 'up' migrations to create tables, data types, indexes etc. The .sql scripts to
do this are located in the migrations folder of your local repository.
ls ~/colouring-core/migrations/*.up.sql 2>/dev/null | while read -r migration; do psql < $migration; done;Load all building outlines.
./load_geometries.sh ./Create a building record per outline.
./create_building_records.shAfter buildings are loaded or their geometries changed, it is necessary to flush cache files so outdated geometries are not shown. This can be achieved by deleting relevant folder at the server.
Now we are ready to run the application.
First enter the app directory.
cd ~/colouring-core/appThen create a folder for the tilecache.
mkdir tilecacheCreate some additional variables for running the application (the APP_COOKIE_SECRET is arbitrary).
export PGPORT=5432
export APP_COOKIE_SECRET=123456
export TILECACHE_PATH=~/colouring-core/app/tilecacheFinally, simply run the application with npm.
npm startNote: You can also specify the variables for npm start like so:
Specify variables
PGPASSWORD=<pgpassword> PGDATABASE=<colouringcitiesdb> PGUSER=<username> PGHOST=localhost PGPORT=5432 APP_COOKIE_SECRET=123456 TILECACHE_PATH=~/colouring-core/app/tilecache npm startThe site can then be viewed on http://localhost:8080 on the host computer.
Finally to quit the application type Ctrl-C.