Export customised OSM data in various file formats
JavaScript Python HTML CSS
Permalink
Failed to load latest commit information.
.tx
api
core
doc
jobs
locales
scripts
tasks
ui
utils
.coveragerc
.gitignore
.travis.yml
LICENSE.md
README.md
auto_clean.bash
manage.py
requirements-dev.txt
requirements.txt
setup.cfg

README.md

Stories in Ready

OSM Export Tool

Osm Export Tool platform allows you to create custom OpenStreetMap exports for various HOT regions. You can specify an area of interest and a list of features (OpenStreetMap tags) for the export. A current OpenStreetMap data extract for that area in various data formats is then created for you within minutes.

This repo contains the newly re-written version 2 of the OSM exports tool. The live site http://export.hotosm.org is now powered by this repo. The older version 1 site is still available at: http://old-export.hotosm.org/ running code from this repo: https://github.com/hotosm/hot-exports

Installation Instructions

Some prior experience with Django would be helpful, but not strictly necessary.

Update Packages

$ sudo apt-get update
$ sudo apt-get upgrade

Python

HOT Exports requires Python 2.7.x.

pip

To install pip, run:

$ sudo apt-get install python-pip

Git

Git is used for version control. To check your version, and if you have it installed:

$ git --version or run $ sudo apt-get install git

Virtualenv

Virtualenv (virtual environment) creates self-contained environments to prevent different versions of python libraries/packages from conflicting with each other.

To make your life easier install virtualenvwrapper

$sudo pip install virtualenvwrapper

Add the following to .bashrc or .profile

export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/dev
source /usr/local/bin/virtualenvwrapper.sh

Run source ~/.bashrc

Run mkvirtualenv hotosm to create the hotosm virtual environment.

Change to the $HOME/dev/hotosm directory and run workon hotosm.

Postgres

Install PostgreSQL / PostGIS and its dependencies,

$ sudo apt-get install libpq-dev python-dev

$ sudo apt-get install postgresql postgresql-contrib

$ sudo apt-get install postgis postgresql-9.3-postgis-2.1

Create the database and role

$ sudo su - postgres
$ createdb 'hot_exports_dev'
$ create role hot with password '<-password->'

You might need to update the pg_hba.conf file to allow localhost connections via tcp/ip or allow trusted connections from localhost.

Run $ psql -h localhost -U hot -W hot_exports_dev

$ ALTER ROLE hot SUPERUSER;
$ ALTER ROLE hot WITH LOGIN;
$ GRANT ALL PRIVILEGES ON DATABASE hot_exports_dev TO hot;
$ CREATE EXTENSION POSTGIS;
$ CREATE EXTENSION HSTORE;

Create the exports schema

$ CREATE SCHEMA exports AUTHORIZATION hot;

Install GDAL

$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
$ sudo apt-get update
$ sudo apt-get install gdal-bin libgdal-dev

To install the python GDAL bindings into your virtualenv you need to tell pip where to find the libgdal header files, so in your shell run:

$ export CPLUS_INCLUDE_PATH=/usr/include/gdal
$ export C_INCLUDE_PATH=/usr/include/gdal

Install third-party dependencies

The HOT Export pipeline depends on a number of third-party tools.

$ sudo apt-get install osmctools

$ sudo apt-get install spatialite-bin libspatialite5 libspatialite-dev

$ sudo apt-get install default-jre zip unzip

Garmin

Download the latest version of the mkgmap utility for making garmin IMG files from http://www.mkgmap.org.uk/download/mkgmap.html

Download the latest version of the splitter utility for splitting larger osm files into tiles. http://www.mkgmap.org.uk/download/splitter.html

Create a directory and unpack the mkgmap and splitter archives into it.

OSMAnd OBF

For details on the OSMAnd Map Creator utility see http://wiki.openstreetmap.org/wiki/OsmAndMapCreator

Download the OSMAnd MapCreator from http://download.osmand.net/latest-night-build/OsmAndMapCreator-main.zip. Unpack this into a directory somewhere.

Install RabbitMQ

HOT Exports depends on the rabbitmq-server. For more detailed installation instructions see http://www.rabbitmq.com/install-debian.html. The default configuration should be fine for development purposes.

$ sudo apt-get install rabbitmq-server

Checkout the HOT Export Tool source

In the hotosm project directory run:

$ git clone git@github.com:hotosm/osm-export-tool2.git

Install the project's python dependencies

From the project directory, install the dependencies into your virtualenv:

$ pip install -r requirements-dev.txt

or

$ pip install -r requirements.txt

Project Settings

Create a copy of core/settings/dev_dodobas.py and update to reflect your development environment.

Look at core/settings/project.py and make sure you update or override the following configuration variables in your development settings:

EXPORT_STAGING_ROOT = 'path to a directory for staging export jobs'

EXPORT_DOWNLOAD_ROOT = 'path to a directory for storing export downloads'

EXPORT_MEDIA_ROOT = '/downloads/' (map this url in your webserver to EXPORT_DOWNLOAD_ROOT to serve the exported files)

OSMAND_MAP_CREATOR_DIR = 'path to directory where OsmAndMapCreator is installed'

GARMIN_CONFIG = 'absolute path to utils/conf/garmin_config.xml'

OVERPASS_API_URL = 'url of your local overpass api endpoint (see Overpass API below)'

Update the utils/conf/garmin_config.xml file. Update the garmin and splitter elements to point to the absolute location of the mkgmap.jar and splitter.jar utilites.

Once you've got all the dependencies installed, run ./manage.py migrate to set up the database tables etc.. Then run ./manage.py runserver to run the server. You should then be able to browse to http://localhost:8000/

Overpass API

The HOT Exports service uses a local instance of Overpass v07.52 for data extraction. Detailed instructions for installing Overpass are available here.

Download a (latest) planet pbf file from (for example) http://ftp.heanet.ie/mirrors/openstreetmap.org/pbf/.

If you're doing development you don't need the whole planet so download a continent or country level extract from http://download.geofabrik.de/, and update the osmconvert command below to reflect the filename you've downloaded.

To prime the database we've used osmconvert as follows:

osmconvert --out-osm planet-latest.osm.pbf | ./update_database --meta --db-dir=$DBDIR --flush-size=1

If the dispatcher fails to start, check for, and remove osm3s_v0.7.52_osm_base from /dev/shm.

We apply minutely updates as per Overpass installation instructions, however this is not strictly necessary for development purposes.

Celery Workers

HOT Exports depends on the Celery distributed task queue. As export jobs are created they are pushed to a Celery Worker for processing. At least two celery workers need to be started as follows:

From a 'hotosm' virtualenv directory (use screen), run:

export DJANGO_SETTINGS_MODULE=core.settings.your_settings_module

$ celery -A core worker --loglevel debug --logfile=celery.log.

This will start a celery worker which will process export tasks. An additional celery worker needs to be started to handle purging of expired unpublished export jobs. From another hotosm virtualenv terminal session in the project top-level directory, run:

export DJANGO_SETTINGS_MODULE=core.settings.your_settings_module

$ celery -A core beat --loglevel debug --logfile=celery-beat.log

See the CELERYBEAT_SCHEDULE setting in core/settings/celery.py.

For more detailed information on Celery Workers see here

For help with daemonizing Celery workers see here

Using Transifex service

To work with Transifex you need to create ~/.transifexrc, and modify it's access privileges

chmod 600 ~/.transifexrc

Example .transifexrc file:

[https://www.transifex.com]
hostname = https://www.transifex.com
password = my_super_password
token =
username = my_transifex_username

Managing source files

To update source language (English) for Django templates run:

python manage.py makemessages -l en

To update source language for javascript files run:

python manage.py makemessages -d djangojs -l en

then, push the new source files to the Transifex service, it will overwrite the current source files

tx push -s

Pulling latest changes from the Transfex

When adding a new language, it's resource file does not exist in the project, but it's ok as it will be automatically created when pulling new translations from the service. To add a local mapping:

tx set -r osm-export-tool2.master -l hr locales/hr/LC_MESSAGES/django.po

or for javascript files:

tx set -r osm-export-tool2.djangojs -l hr locales/hr/LC_MESSAGES/djangojs.po

Once there are some translation updates, pull the latest changes for mapped resources

For a specific language(s):

tx pull -l fr,hr

For all languages:

tx pull

Finally, compile language files

python manage.py compilemessages