Osm Export Tool platform allows you to create custom OpenStreetMap exports for various HOT regions. You can specify an area of interest and a list of features (OpenStreetMap tags) for the export. A current OpenStreetMap data extract for that area in various data formats is then created for you within minutes.
This repo contains the newly re-written version 2 of the OSM exports tool. The live site http://export.hotosm.org is now powered by this repo. The older version 1 site is still available at: http://old-export.hotosm.org/ running code from this repo: https://github.com/hotosm/hot-exports
Some prior experience with Django would be helpful, but not strictly necessary.
$ sudo apt-get update $ sudo apt-get upgrade
HOT Exports requires Python 2.7.x.
To install pip, run:
$ sudo apt-get install python-pip
Git is used for version control. To check your version, and if you have it installed:
$ git --version
or run $ sudo apt-get install git
Virtualenv (virtual environment) creates self-contained environments to prevent different versions of python libraries/packages from conflicting with each other.
To make your life easier install virtualenvwrapper
$sudo pip install virtualenvwrapper
Add the following to .bashrc
or .profile
export WORKON_HOME=$HOME/.virtualenvs export PROJECT_HOME=$HOME/dev source /usr/local/bin/virtualenvwrapper.sh
Run source ~/.bashrc
Run mkvirtualenv hotosm
to create the hotosm virtual environment.
Change to the $HOME/dev/hotosm
directory and run workon hotosm
.
Install PostgreSQL / PostGIS and its dependencies,
$ sudo apt-get install libpq-dev python-dev
$ sudo apt-get install postgresql postgresql-contrib
$ sudo apt-get install postgis postgresql-9.3-postgis-2.1
$ sudo su - postgres $ createdb 'hot_exports_dev' $ create role hot with password '<-password->'
You might need to update the pg_hba.conf
file to allow localhost connections via tcp/ip or
allow trusted connections from localhost.
Run $ psql -h localhost -U hot -W hot_exports_dev
$ ALTER ROLE hot SUPERUSER; $ ALTER ROLE hot WITH LOGIN; $ GRANT ALL PRIVILEGES ON DATABASE hot_exports_dev TO hot; $ CREATE EXTENSION POSTGIS; $ CREATE EXTENSION HSTORE;
Create the exports schema
$ CREATE SCHEMA exports AUTHORIZATION hot;
$ sudo apt-get install software-properties-common $ sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable $ sudo apt-get update $ sudo apt-get install gdal-bin libgdal-dev
To install the python GDAL bindings into your virtualenv you need to tell pip where to find the libgdal header files, so in your shell run:
$ export CPLUS_INCLUDE_PATH=/usr/include/gdal $ export C_INCLUDE_PATH=/usr/include/gdal
The HOT Export pipeline depends on a number of third-party tools.
$ sudo apt-get install osmctools
$ sudo apt-get install spatialite-bin libspatialite5 libspatialite-dev
$ sudo apt-get install default-jre zip unzip
Download the latest version of the mkgmap utility for making garmin IMG files from http://www.mkgmap.org.uk/download/mkgmap.html
Download the latest version of the splitter utility for splitting larger osm files into tiles. http://www.mkgmap.org.uk/download/splitter.html
Create a directory and unpack the mkgmap
and splitter
archives into it.
For details on the OSMAnd Map Creator utility see http://wiki.openstreetmap.org/wiki/OsmAndMapCreator
Download the OSMAnd MapCreator from http://download.osmand.net/latest-night-build/OsmAndMapCreator-main.zip. Unpack this into a directory somewhere.
HOT Exports depends on the rabbitmq-server. For more detailed installation instructions see http://www.rabbitmq.com/install-debian.html. The default configuration should be fine for development purposes.
$ sudo apt-get install rabbitmq-server
In the hotosm project directory run:
$ git clone git@github.com:hotosm/osm-export-tool2.git
From the project directory, install the dependencies into your virtualenv:
$ pip install -r requirements-dev.txt
or
$ pip install -r requirements.txt
Create a copy of core/settings/dev_dodobas.py
and update to reflect your development environment.
Look at core/settings/project.py
and make sure you update or override the following configuration variables in your development settings:
EXPORT_STAGING_ROOT = 'path to a directory for staging export jobs'
EXPORT_DOWNLOAD_ROOT = 'path to a directory for storing export downloads'
EXPORT_MEDIA_ROOT = '/downloads/' (map this url in your webserver to EXPORT_DOWNLOAD_ROOT to serve the exported files)
OSMAND_MAP_CREATOR_DIR = 'path to directory where OsmAndMapCreator is installed'
GARMIN_CONFIG = 'absolute path to utils/conf/garmin_config.xml'
OVERPASS_API_URL = 'url of your local overpass api endpoint (see Overpass API below)'
Update the utils/conf/garmin_config.xml
file. Update the garmin
and splitter
elements to point to the
absolute location of the mkgmap.jar
and splitter.jar
utilites.
Once you've got all the dependencies installed, run ./manage.py migrate
to set up the database tables etc..
Then run ./manage.py runserver
to run the server.
You should then be able to browse to http://localhost:8000/
The HOT Exports service uses a local instance of Overpass v07.52 for data extraction. Detailed instructions for installing Overpass are available here.
Download a (latest) planet pbf file from (for example) http://ftp.heanet.ie/mirrors/openstreetmap.org/pbf/.
If you're doing development you don't need the whole planet so download a continent or country level extract from http://download.geofabrik.de/,
and update the osmconvert
command below to reflect the filename you've downloaded.
To prime the database we've used osmconvert
as follows:
osmconvert --out-osm planet-latest.osm.pbf | ./update_database --meta --db-dir=$DBDIR --flush-size=1
If the dispatcher fails to start, check for, and remove osm3s_v0.7.52_osm_base
from /dev/shm
.
We apply minutely updates as per Overpass installation instructions, however this is not strictly necessary for development purposes.
HOT Exports depends on the Celery distributed task queue. As export jobs are created they are pushed to a Celery Worker for processing. At least two celery workers need to be started as follows:
From a 'hotosm' virtualenv directory (use screen), run:
export DJANGO_SETTINGS_MODULE=core.settings.your_settings_module
$ celery -A core worker --loglevel debug --logfile=celery.log
.
This will start a celery worker which will process export tasks. An additional celery worker needs to be started to handle purging of expired unpublished export jobs. From another hotosm virtualenv terminal session in the project top-level directory, run:
export DJANGO_SETTINGS_MODULE=core.settings.your_settings_module
$ celery -A core beat --loglevel debug --logfile=celery-beat.log
See the CELERYBEAT_SCHEDULE
setting in core/settings/celery.py
.
For more detailed information on Celery Workers see here
For help with daemonizing Celery workers see here
To work with Transifex you need to create ~/.transifexrc
, and modify it's access privileges
chmod 600 ~/.transifexrc
Example .transifexrc
file:
[https://www.transifex.com]
hostname = https://www.transifex.com
password = my_super_password
token =
username = my_transifex_username
To update source language (English) for Django templates run:
python manage.py makemessages -l en
To update source language for javascript files run:
python manage.py makemessages -d djangojs -l en
then, push the new source files to the Transifex service, it will overwrite the current source files
tx push -s
When adding a new language, it's resource file does not exist in the project, but it's ok as it will be automatically created when pulling new translations from the service. To add a local mapping:
tx set -r osm-export-tool2.master -l hr locales/hr/LC_MESSAGES/django.po
or for javascript files:
tx set -r osm-export-tool2.djangojs -l hr locales/hr/LC_MESSAGES/djangojs.po
Once there are some translation updates, pull the latest changes for mapped resources
For a specific language(s):
tx pull -l fr,hr
For all languages:
tx pull
Finally, compile language files
python manage.py compilemessages