Skip to content

Installation Guide

Jump to bottom Edit New page
Andrew Thompson edited this page Jul 25, 2014 · 41 revisions

If your intention is to set up OTM2 for development and contributing to the open source project, instead of using this guide, we recommend you use the otm2-vagrant project instead of following these steps. That will get you a "dev env" much more quickly!

This page is a work in progress and will be updated (mind the TODOs for now). This guide assumes use of 64-bit Ubuntu 12.04. IMPORTANT: OpenTreeMap2 will ONLY work on 64-bit Ubuntu. Do not attempt with 32-bit.

There are several config files, upstart scripts, and other small files that you'll need to copy/get from the otm2-vagrant project's "scripts" directory. We'll link to them here when you need them.

Step 1: Install basics.

The vast majority of commands in this guide require superuser permissions. If you're comfortable, go ahead and sudo su to change to the root user to save yourself some repetitive sudo'ing.

apt-get update
apt-get install python-setuptools python-software-properties
easy_install pip #the pip Python package manager must be installed with easy_install! Important!
pip install virtualenv #then the virtualenv package must be installed with pip, also important
apt-get install git

Step 2: Install the PostgreSQL database and PostGIS

apt-get install postgresql postgresql-server-dev-9.1 postgresql-contrib
apt-get install build-essential libxml2-dev libgeos-dev libproj-dev libgdal1-dev libpq-dev python-dev

#download and compile PostGIS:

mkdir -p /usr/local/postgis/download
cd /usr/local/postgis/download
wget http://postgis.org/download/postgis-2.0.1.tar.gz
tar -xzf postgis-2.0.1.tar.gz
cd postgis-2.0.1
./configure --without-topology
make
make install

Now we need to restart Postgres, and and create a user, database, and extensions for OTM2 to use. This will need to be done in the psql prompt as the "postgres" system user.

service postgresql restart
su postgres
psql

The following commands should be done in the psql prompt:

CREATE USER pick_an_otm_username SUPERUSER PASSWORD ‘pick_a_password’; --Use superuser for django tests to work, but don’t use this if you're not building a development environment. Otherwise, use “ENCRYPTED” in place of "SUPERUSER".
\c template1; --connect to the template database to add the hstore extension
CREATE EXTENSION IF NOT EXISTS hstore; --must be created in template1
CREATE DATABASE pick_an_otm_db_name OWNER pick_an_otm_username; --the actual database 
\c pick_an_otm_db_name
CREATE EXTENSION IF NOT EXISTS postgis; --The PostGIS extension *must* be created in the OTM database, and NOT the template1 database. Django tests try to create postgis in the template db and will fail if it’s already there
\q --quit the psql prompt

Now, exit from the postgres user back to root or your system user:

exit

Step 3: Install the nginx HTTP server

apt-get install nginx

Create/add an /etc/nginx/nginx.conf file based on the default in the otm2-vagrant project If you're aiming for a production environment, you may want to add a proxy_cache_path line similar to the below:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=one:300m inactive=21600m;

We'll also need to create/add an /etc/nginx/sites-available/otm.conf file like the default in the otm2-vagrant project. It should define a server for the OTM app and some other details like the below:

include includes/upstreams/*;

server {
  listen 80 default_server;
  
  client_max_body_size 20M;

  include includes/locations/*;
}

Once we've added the file in the sites-available directory, we need to symlink it to the sites-enabled directory to enable it to be served by nginx:

cd /etc/nginx
rm sites-enabled/default
ln -s /etc/nginx/sites-available/otm.conf /etc/nginx/sites-enabled/otm.conf #must provide the full paths here for symlinking to work right

Now we need to create some directories for the later steps.

mkdir -p /etc/nginx/includes/upstreams
mkdir -p /etc/nginx/includes/locations
mkdir /var/cache/nginx
chown www-data:www-data /etc/nginx/includes
chown www-data:www-data /etc/nginx/includes/upstreams
chown www-data:www-data /etc/nginx/includes/locations
chown www-data:www-data /var/cache/nginx

Step 4: Install the ecobenefits project

The ecobenefits repo is written in the Go language, so we'll need to install Go as well.

apt-get install gettext libfreetype6-dev libjpeg-dev zlib1g-dev libpq-dev python-dev libxml2-dev libgeos-dev libproj-dev libgdal1-dev build-essential #most of these aside from the first two should already be installed

#link some of the libraries
ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib/libjpeg.so
ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib/libfreetype.so
ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/libz.so

Now we'll go about downloading and setting up Go v1.2:

wget "https://go.googlecode.com/files/go1.2.linux-amd64.tar.gz" -O /tmp/go.tar.gz
tar -C /usr/local -xzf /tmp/go.tar.gz

export PATH=$PATH:/usr/local/go/bin #add go to the PATH

You'll have to change your /etc/profile file and add the following line for a system-wide install:

export PATH=$PATH:/usr/local/go/bin

Now we'll compile Go from source, and build the ecoservice project:

cd /usr/local/go/src
./all.bash

cd /usr/local
mkdir ecoservice

export GOPATH=/usr/local/ecoservice

go get github.com/OpenTreeMap/ecobenefits # -v flag for verbose optional
go build github.com/OpenTreeMap/ecobenefits

Create/add a /usr/local/ecoservice/config.gcfg file like the default in the otm2-vagrant project with similar information to the below. Make sure you update your values for database user, name, and password:

[database]
user=otm                
database=otm                
password=otm                    
host=localhost           

[server]
host=127.0.0.1
port=13000        

[data]
path=/usr/local/ecoservice/data/

Now, we'll have to give the ecoservice Go code the data to use, and put it in the directory we just defined in the config.gcfg file:

mkdir /usr/local/tmp
cd tmp
git clone https://github.com/OpenTreeMap/ecobenefits.git
cp -r ecobenefits/data /usr/local/ecoservice/ #copy data dir over

We'll need to add an upstart script for the ecoservice like the default in the otm2-vagrant project. Similar information to the below in a file like /etc/init/ecoservice.conf should do it:

description "Eco benefit service"

start on runlevel [2345]
stop on runlevel [!2345]

respawn
respawn limit 50 5

kill timeout 5

chdir /usr/local/ecoservice

exec ./ecobenefits -configpath=./config.gcfg 2>&1 1>/var/log/otm-eco.log

Step 5: Install Django and the OTM2 Django apps

Now we'll clone the OTM2 repo, make a virtualenv for it, and install the requirements (for production, development, and test) in that environment.

mkdir /usr/local/otm
cd /usr/local/otm
git clone https://github.com/OpenTreeMap/OTM2.git
mv OTM2 app
virtualenv env
source env/bin/activate
cd app
pip install -r requirements.txt
pip install -r dev-requirements.txt
pip install -r test-requirements.txt

We'll need to create/add and edit a /usr/local/otm/app/opentreemap/opentreemap/local_settings.py file like the default in the otm2-vagrant project. A production local_settings.py file can have a lot in it, but the important points are detailed below:

TODO spruce this example up

EMAIL_HOST = ‘localhost’
EMAIL_PORT = ‘25’
PROJECT_ROOT = ‘/usr/local/otm/app/opentreemap’
MEDIA_ROOT = ‘/usr/local/otm/media/’
DATABASES = { ‘default’: { …. ‘HOST’: ‘localhost’,}} #you’ll need to add your db’s info
BROKER_URL = ‘redis://localhost:6379/’
CELERY_RESULT_BACKEND = ‘redis://localhost:6379/’
STATIC_ROOT = ‘/usr/local/otm/static/’
#all “_log_level” vars to DEBUG for production envs
# django_log to ‘/var/log/otm-django.log’
# django_request_log to ‘/var/log/otm-django-request.log’
# celery_log to ‘/var/log/otm-celery.log’

Now, we'll have to use south to sync and apply the OTM database migrations.

cd /usr/local/otm/app/opentreemap #the folder with manage.py in it
python manage.py syncdb
python manage.py migrate

Step 6: Run and test the Ecobenefits service

We needed to run the "syncdb" command above to apply the database migrations before the ecobenefits service had the information it needs to start. Let's give it a shot now!

cd /usr/local
./ecobenefits -configpath=./ecoservice/config.gcfg

Now, visit a page like http://localhost:13000/eco.json?otmcode=FICA&instanceid=1&speciesid=1&diameter=20&region=InlEmpCLM (either with a browser or something like wget or curl), and you should get some JSON output describing the ecosystem benefits for a single tree! Congrats, this part works!

Step 7: Set up the unicorn upstart script

We're using Gunicorn as an app server, and we'll need an upstart script for it too like the default in the otm2-vagrant project. Put it in something like /etc/init/otm-unicorn.conf, check for the below:

chdir /usr/local/otm/app/opentreemap
exec /usr/local/otm/env/bin/gunicorn -w 2 --log-file "/var/log/otm-unicorn.log" -b "0.0.0.0:12000" opentreemap.wsgi:application

Note there's only two lines there - the second line's "opentreemap.wsgi:application" may wrap to a third depending on how you're viewing these instructions, but it's definitely just a space and not a newline!

Step 8: Run Django's collectstatic static files command

cd /usr/local/otm/app/opentreemap
python manage.py collectstatic #with virtualenv activated

Step 9: Set up the celery upstart script

We'll need to create/add another upstart script for the celery daemon - put it in /etc/init/celeryd.conf like the default in the otm2-vagrant project. The key line to check with your install is below:

exec /usr/local/otm/env/bin/python /usr/local/otm/app/opentreemap/manage.py celeryd

Step 10: Create log files if they don't exist yet

We'll do a quick listing of the log directory and pipe it through grep to see which log files were already created.

ls -al /var/log | grep "otm"

We need these 5 files:

  • /var/log/otm-celery.log
  • /var/log/otm-django.log
  • /var/log/otm-django-request.log
  • /var/log/otm-eco.log
  • /var/log/otm-unicorn.log

See any that are missing from what your 'ls' command showed you and the list above? Add them with:

touch /var/log/otm-whatever.log

Now, we need to make sure the files are owned by your main Ubuntu user account (ie, not root):

chown your_user_here:your_user_here /var/log/otm-celery.log
chown your_user_here:your_user_here /var/log/otm-django.log
chown your_user_here:your_user_here /var/log/otm-django-request.log
chown your_user_here:your_user_here /var/log/otm-eco.log
chown your_user_here:your_user_here /var/log/otm-unicorn.log

Step 11: Add and edit the nginx includes

We need to add two more files for nginx - akin to the defaults in otm2-vagrant for /etc/nginx/includes/upstreams/app and /etc/nginx/includes/locations/app.

Step 12: Restart nginx and the gunicorn servers and test

service nginx restart
service otm-unicorn restart

Test it! Go to localhost in a browser (or localhost:12000) and you should see the basic beginnings of an OpenTreeMap page!

Step 13: Install the OTM2-tiler map tiler project

For the OTM2-tiler project, we're going to need to add a few aptitude repositories, apt-get install a bunch of dependencies, and get and install Node.JS.

First, the repositories:

add-apt-repository ppa:mapnik/boost
add-apt-repository ppa:mapnik/v2.1.0

Second, install the dependencies:

apt-get install libboost-dev libboost-filesystem-dev libboost-program-options-dev libboost-python-dev libboost-regex-dev libboost-system-dev libboost-thread-dev

apt-get install libsigc++-2.0-dev libmapnik-dev mapnik-utils

Third, get and untar the right version (v0.10.26 works well) of Node.JS:

cd /usr/local
mkdir /usr/local/node
wget http://nodejs.org/dist/v0.10.26/node-v0.10.26-linux-x64.tar.gz -O node.tar.gz
tar xvf node.tar.gz --strip-components=1 -C /usr/local

The tar command above will create the following files and directories in /usr/local:

  • ChangeLog
  • LICENSE
  • Readme.md
  • include/node
  • lib/dtrace
  • lib/node
  • lib/node_modules
  • share/man/man1/node.1

Fourth, we need to grab the OTM2-tiler code from Github:

cd /usr/local
git clone https://github.com/OpenTreeMap/OTM2-tiler.git
mv OTM2-tiler tiler #change the cloned directory name just to make it simpler
cd tiler

Fifth, we need to create a settings.json file like it says in the OTM2-tiler readme. These values may be similar or different to yours:

{
    "username": "user",
    "host": "127.0.0.1",
    "password": "users_password",
    "port": "5432",
    "redishost": "127.0.0.1"
}

Sixth, run the node package manager. This one command should get all the node libraries the tiler needs and configure them:

npm install

Step 14: Add tiler upstart script

Another upstart script! This one should be something like the /etc/init/tiler.conf in the otm2-vagrant project and has one line of note to check with your install:

chdir /usr/local/tiler

Step 16: Add nginx includes for the tiler

Similar to step 11, we need to add two more include files to show nginx where our tiler is - /etc/nginx/includes/locations/tiler and /etc/nginx/includes/upstreams/tiler. The otm2-vagrant project has defaults for locations/tiler and upstreams/tiler.

Step 17: Install a mail server

Sendmail tends to work well:

apt-get install sendmail

Step 18: Install and configure the Redis server

apt-get install redis-server

And it's a good idea if you're moving to production to add a logrotate config for redis at /etc/logrotate.d/redis too.

After creating the last file, make sure it's properly owned by root:

chown root:root /etc/logrotate.d/redis
chmod 0644 /etc/logrotate.d/redis

Step 19: (optional, for testing/dev) Install dependencies for running the selenium tests

Make sure you already installed the test-requirements.txt file previously!

apt-get install firefox
apt-get install xvfb

Step 20: Create an OTM system user

cd /usr/local/otm/app/opentreemap
#with your virtualenv activated (source env/bin/activate)
python manage.py create_system_user

#Step 21: Fabric, grunt, and npm

"fab me static" will run grunt and npm to collect/compile all the front-end CSS/JS/etc. If you're not already root, you'll need a password. The "me" option makes fabric use the localhost (ie, we're not deploying to a remote machine).

pip install fabric
fab me static

Next, install and use grunt:

npm install -g grunt-cli
cd /usr/local/otm/app #this is wherever the package.json file lives that grunt uses
npm install
python manage.py collectstatic

#Step 22: Create an OTM instance!

You'll need to use manage.py's create_instance command. TODO - right now, there are undocumented options for this command that need to be fixed (some were in #1414 ) (all options are in "-h"...bug in this, branch merge is pending)

python manage.py create_instance

#Step 23: Restart all services!

service nginx restart
service otm-unicorn restart
service sendmail restart
service redis-server restart

With any luck, your OpenTreeMap install should be running on localhost!

Add a custom footer
Add a custom sidebar
Clone this wiki locally