The server application for the WalkersGuide project: A navigation aid for visual impaired users
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
PublicTransportInterface
misc
scripts
sql_functions
src
CHANGES
LICENSE
readme

readme

WalkersGuide-Server
===================

Introduction
------------

WalkersGuide is a navigation aid based on OpenStreetMap data. It's designed especially but not
exclusively for visual impaired people. It's main purpose is to calculate routes for walkers
including public transportation. More about this project at http://walkersguide.org.

The project consists of two parts:
1. The server holds the OpenStreetMap data and calculates routes based on given start and
destination. It also provides an interface to query nearby points of interest.
2. The client sends start and destination points to the server. After calculation the received route
is presented to the user.

This repository holds the server application. The following section gives an overview of the
project structure. The subsequent sections contain installation instructions and usage notes.

Project structure
-----------------

The map data for the routing process comes from [OpenStreetMap](http://openstreetmap.org). A country
or continent is downloaded and stored in a local Postgresql database. The scripts from the
"sql_functions" and "scripts" folders help you to get a local copy of the OpenStreetMap database on
your server. But they not only create the database but also calculate additional database tables
like intersections, poi and a routing graph. These tables are required to perform the creation of a
walkers route.

The "src" folder contains python scripts to query data from the database and calculate the actual
route. There is also a web server, which listens for client requests on a specific port, calls the
route creation functions and returns the results to the client.

The PublicTransportInterface folder holds a small Java program to query public transportation data
like routes and departure timetables. The program serves as a wrapper for the Java library
[public-transport-enabler](https://github.com/schildbach/public-transport-enabler). This library
fetches data from the German public transportation provider "Deutsche Bahn". Therefore the public
transportation functionality currently is limited outside of Germany.

Installation
------------

This section describes the installation process of the required software. The instructions cover the
Debian operating system.

### Postgresql, Postgis and pgrouting ###

We need Postgresql version >= 9.1, Postgis version >= 2.0 and pgrouting.  If you still use Debian
Wheezy, you must add the external Postgresql software repository first. That's necessary cause the
Debian repositories still contain version 1.5 of the Postgresql extension Postgis but at least v2.0
is required. If you use Debian Jessie instead, you can skip the following instructions.

```
root# echo "deb http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg main" > /etc/apt/sources.list.d/pgdg.list
root# wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc |  apt-key add -
root# aptitude update && aptitude upgrade
```

Then install Postgresql, Postgis (an extension to handle spacial data types like points and lines)
and pgrouting. You can choose every Postgresql version since 9.1:

```
root# aptitude install postgresql-9.3 postgresql-contrib-9.3 postgresql-9.3-postgis-2.1 postgresql-9.3-pgrouting
```

### Java ###

Java JDK to compile the wrapper of the public transportation library:

```
root# aptitude install openjdk-7-jdk ant
```

### Python plugins ###

The WalkersGuide-Server project requires some python modules, which should be installed next:

```
root# aptitude install python-setuptools
root# easy_install pip
osm$ pip --user --upgrade install py4j requests configparser cherrypy psycopg2
```

### Webserver installation ###

The WalkersGuide Android client since version 0.4.0 requires a secure connection via SSL. Don't use
cherrypy's own SSL server for that but Install and configure a webserver like nginx instead. The
webserver should handle the SSL connection and redirect traffic to the WalkersGuide server
component, which only runs locally at a different port. You can find a sample configuration for nginx
at misc/nginx-walkersguide.org.conf.

### configure mail sending ###

Some later used scripts send mails to inform the user about their status. This is done via the mail
command of the server. If it's not already configured to send mail, it can be done the following
way:

```
root# aptitude install mailutils msmtp msmtp-mta
```

The package "mailutils" provides the "mail" command. The mail is send via msmtp and msmtp-mta makes
msmtp the default mail transfer agent. Now configure msmtp by creating the file .msmtprc in the
users home directory with the following content:

```
defaults
tls on
tls_starttls on
auth on
logfile ~/.msmtp.log
syslog off
# smtp of your mail server
account my_account
    host smtp.example.org
    port 587
    from sender@example.org
    tls_certcheck off
    user my_user_name
    password my_password
account default my_account
```

Make the file only readable for the current user:

```
$ chmod 600 ~/.msmtprc
```

Now you can send a mail with the following command:

```
$ echo "my message body" | mail -s "my subject" recipient@example.org
```

### Other required programs ###

Create a folder for the project, for example:

```
mkdir /data/routing
cd /data/routing
```

Then create a tools folder for some additional helper programs:

```
mkdir tools
cd tools
```

Download Osmosis, a program to extract parts of a OpenStreetMap database:

```
wget http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.zip
unzip osmosis-latest.zip -d osmosis
rm osmosis-latest.zip
```

OSMFilter and OSMConvert:

```
wget -O - http://m.m.i24.cc/osmfilter.c |cc -x c - -O3 -o osmfilter
wget -O - http://m.m.i24.cc/osmconvert.c | cc -x c - -lz -O3 -o osmconvert
```

Osm2po creates a database table which represents the routing graph of all streets and ways. This
graph is needed by pgrouting to calculate a route from start to the point of destination. The tool
must be downloaded from http://www.osm2po.de/ and extracted to the folder named "osm2po".

To use osm2po you have to  accept it's license once. To do so, enter it's folder, start the
demo.sh script and type "yes" when instructed. After that you can cancel the process and delete the
already created folder.

```
cd osm2po
chmod +x demo.sh
./demo.sh
# type yes and cancel by pressing ^c
rm -R hh
```

Lastly clone this repository:

```
cd ..
git clone https://github.com/scheibler/WalkersGuide-Server.git
```

If you plan to run multiple server instances for example to offer maps of several countries or
continents, you also can clone in another folder:

```
cd ..
git clone https://github.com/scheibler/WalkersGuide-Server.git WalkersGuide-Europe
```

After this, the installation process is finished. Next follows the configuration.


Initial Configuration
---------------------

The first part of this section covers the configuration of the Postgresql server. Later on follows
the configuration of the WalkersGuide-Server package.

### Postgresql ###

Create a new database user first and assign a password. It must be a super user:

```
root# su postgres
postgres$ createuser -P -S wgs_writer
postgres$ exit
```

The option for superuser also could be "-s".

After that, change database access permissions in the file

```
root# sudo vim /etc/postgresql/9.3/main/pg_hba.conf
```

to the following ones:

```
# postgres system user can log into all databases
local   all             postgres                                peer
# disable peer login
# that means, that the current console user can't access databases anymore
#host    all             all             127.0.0.1/32            md5
#host    all             all             ::1/128                 md5
# give the above created database user permission, to access the WalkersGuide osm databases
host    all             wgs_writer             127.0.0.1/32            md5
```

Now you have to change some settings in the Postgresql main config. The defaults are fairly
conservative and often don't fit the needs for a large db. The settings hardly depend on the
hardware of your server. These are mine for a server with a Core I7 CPU, 32 GB Ram and 3 TB HDD:

```
root# vim /etc/postgresql/9.3/main/postgresql.conf
[...]
#------------------------------------------------------------------------------
# CUSTOMIZED OPTIONS
#------------------------------------------------------------------------------
effective_cache_size = 22GB # pgtune wizard 2014-07-01
shared_buffers = 9GB
maintenance_work_mem = 1GB # pgtune wizard 2014-07-01
work_mem = 256MB   # suggested 192MB
temp_buffers = 32MB
wal_buffers = 16MB   # suggested 8MB
checkpoint_segments = 64   # suggested 16
checkpoint_completion_target = 0.9 # pgtune wizard 2014-07-01
default_statistics_target = 500   # suggested 50
constraint_exclusion = on # pgtune wizard 2014-07-01
max_connections = 20 # pgtune wizard 2014-07-01
random_page_cost = 2.0

# encoding and localisation
datestyle = 'iso, dmy'
lc_messages = 'de_DE.UTF-8'
lc_monetary = 'de_DE.UTF-8'
lc_numeric = 'de_DE.UTF-8'
lc_time = 'de_DE.UTF-8'
default_text_search_config = 'pg_catalog.german'
```

The suggested values come from pgtune, a Postgresql settings parameters optimizer. You can try it
out yourself easily:

```
root# aptitude install pgtune
root# pgtune -i /etc/postgresql/9.3/main/postgresql.conf -o /tmp/postgresql.conf.tuned
```

Lastly restart Postgresql:

```
root# service postgresql restart
```

To speed up large data imports, it's recommended to use slightly different settings for Postgresql.
Especially AutoVacuum and the bigger MaintenanceWorkMem make the import process much faster. After
you've changed the configuration, you have to restart Postgresql again. Don't forget to reset these
settings to their defaults afterwards.

```
# settings for large data import
maintenance_work_mem = 8GB
autovacuum = off
fsync = off
full_page_writes = off
```

### WalkersGuide-Server config ###

First, enter the project directory and create some folders in it:

```
cd WalkersGuide-Server
mkdir logs
mkdir maps
mkdir tmp
```

Then rename the configuration file in the scripts folder and adapt it's settings to your needs.

```
mv scripts/configuration.sh.example scripts/configuration.sh
```

Afterwards do the same with the config file in the src folder.

```
mv src/configuration.rc.example src/configuration.rc
```

Lastly you have to adapt the settings of the osm2po program. You can find a sample config file for
the osm2po version 5.0 in the misc folder. This file is optimized for the usage with the
WalkersGuide-Server application. In most cases you don't need to change it. Just copy it where you
want to and specify it's path in the configuration.rc file in the scripts directory.


Create new database
-------------------

Now it's time to create the OpenStreetMap database. That's a two step process:
1. Import a country or continent into a newly created Postgresql database.
2. Calculate additional data, which is required for routing.

First, change into the scripts folder:

```
cd scripts
```

### Import OSM database ###

You set the map download URL in the configuration.sh file. Default is the map of Europe, downloaded
from http://geofabrik.de. After setting all necessary configuration options, start the import process with:

```
./osm_database_wrapper.sh create_raw
```

The wrapper script starts "create_raw_database.sh" and creates a so called raw database of the
chosen map area. For big areas like Europe this process can last one or two weeks. After the import
is finished, the newly created database "osm_raw_europe" holds an exact copy of the OpenStreetMap
map.

To speed up the import process, it's recommended to use the different configuration parameters for
Postgresql, which you can find in the previous section of this document.

The OSM data underlies constant changes. The Geofabrik server offers daily differential update files
to keep the local database up to date. Call the wrapper script with the "update_raw" parameter to
start the update process:

```
./osm_database_wrapper.sh update_raw
```

For automatic updates it's recommended to create a Cronjob, which calls the script every night:

```
0  2  *  *  *  /data/routing/WalkersGuide-Server/scripts/osm_database_wrapper.sh update_raw
```

### Create a productive database ###

The next step is to create a second database for productive usage. This includes:
* intersection table with all intersections and their appropriate streets
* tables for poi, stations, buildings, their entrances and traffic signals
* a routing table (eu_2po4pgr)

Start the creation process with the following command:

```
./osm_database_wrapper.sh create_productive
```

The wrapper starts the script "create_productive_database.sh". This script makes a copy of the raw
database and creates the tables mentioned above. During this process, updating of the raw database
is prohibit.

This procedure is recommended for smaller databases but for bigger ones like Europe, the creation
lasts at least two or three days. If the server must process routing queries at the same time, the
query time will be much higher then normal. To prevent the slowdown for the user, it's possible to
calculate the productive database at a second server and only transfer it, when done

To accomplish this, you have to do the following:

1. Follow the installation guide for both servers. Server 1 (the calculation server) also needs
   access over ssh to the server 2 (routing server). Add the respective data to the configuration.sh
   file at server 1 (remote access section).

2. On server 1, you start the creation of the raw and then the productive osm database.

3. Then launch the following script on server 1 to dump the created productive database and transfer
it to server 2 via rsync:

```
./osm_database_wrapper.sh transfer_productive
```

4. After that script finished, switch to server 2 and restore the dump file into the database:

```
./osm_database_wrapper.sh create_productive_from_dumps
```


Usage
-----

When installation and configuration is finished and a productive database is created, you can start
the web server with this command:

```
/data/routing/WalkersGuide-Server/src/webserver.py
```

Make sure, that the given port is reachable from outside. Then activate the public transportation
wrapper:

```
java -jar /data/routing/PublicTransportInterface/dist/PublicTransportInterface.jar [port]
```

If you changed the gateway_port in the file src/configuration.rc you have to specify it at the
program call. Otherwise the public transport interface listens at the standard port 25333. This port
must not be accessible from outside.

Now you can check the web server by opening the following URL at a client, for example under Linux:

```
wget -O - http://hostname:port/get_version | gunzip
```

The server's answer is gzip compressed, so it's necessary to unpack it, before you can view the resulting
data. Answer should look like:

```
{"interface": 1, "warning": "", "map_version": "2014-07-18", "server": "0.1.0", "error": ""}
```