regular-routes-server

Operator of an old regularroutes server BEWARE!!! The master-branch now defaults to PostgreSQL version 10. If you are running a server on 9.x, the databases between PostgreSQL major versions are not compatible. Please check the migration instructions for guidance on how to check your database version and consider migrating to the latest versions. If you don't, use the chef12_fix branch instead of master as server_branch in your regularroutes-srvr.json.

Introduction

"Learning Regular Routes" -server project contains the server-side code for the "learning regular routes"-project. The server is currently split into four separate components:

1. regularroutes-site: An end-user website supporting oauth2 login with a Google account. Offered functionality includes display of energy certificate, display of daily trips on a map, summary and editing of recognised transport modes, download of own data and cancellation of participation. Accessible from the root address of the site.
2. regularroutes-api: Interface to the mobile client.
3. regularroutes-dev: Developer operations, not to be left openly accessible on a production site. Currently includes visualizations (both current data and predicted points) based on client numbers and generation of CSV-dumps from the data.
4. regularroutes-scheduler: Scheduled operations. Currently includes device_data filtering, trip and destination recognition, retrieval of mass transit live data and computation of statistics for energy certificate generation.

Setting up a new TrafficSense client-server pair

This is the procedure for setting up a new client-server pair to run the TrafficSense service.

1. Server setup: Install and initialize the server as detailed in the regular-routes-devops readme. The chef initialisation script used by the procedure in devops clones the designated branch of this repository.
2. Client setup: Build a corresponding client as detailed in the trafficsense-android readme.

Setting up a new server development environment

This is the procedure to set up an environment for developing server software on any local machine.

1. Since the addition of python libraries used by prediction (numpy, scipy etc.), some extra installations are required for the successful installation of the python libraries. On Ubuntu: $ sudo apt-get install build-essential python3-pip python3-dev python3-venv gfortran libatlas-base-dev libblas-dev liblapack-dev libssl-dev. On Mac OS X others are included in Xcode, but a fortran compiler is required. It belongs to the gcc package, on brew: $ brew install gcc.

2. If using a local database, install postgresql and postgis. Available for Debian Linuxes (Ubuntu) via apt-get, and homebrew on Mac OS X. An alternative would be to tunnel onto a database on another server, but the installation of the psycopg2 python library below fails if pg_config, a part of postgresql, is not available.

3. Create a regularroutes database - empty or populated - as [instructed] (https://github.com/aalto-trafficsense/regular-routes-server/tree/master/sql_admin). Leave a postgresql server running (postgres -D rrdb, where rrdb is the path to the directory with your database). If the postgresql server is on another machine, arrange tunnels accordingly. One way or another you are going to need "waypoints" in the db. If you have a local tar available, load it with pg_restore -U postgres -W -d postgres my_waypoints.tar.

4. Clone this (regular-routes-server) repo to a directory where development is to be carried out. Go (cd) into your repository root directory and start a local branch (git checkout -b my_test) if practical.

5. Create a regularroutes.cfg file (it is listed in .gitignore, so shouldn't spread out to github) with the following contents:

      SECRET_KEY = 'secretkey'
      SQLALCHEMY_DATABASE_URI = 'postgresql+psycopg2://regularroutes:qwerty@localhost/postgres'
      MAPS_API_KEY = 'INSERT_Browser_API_key_from_Google_Developers_Console'
      FIREBASE_KEY = 'INSERT_Server_key_from_Firebase_Console'
      RR_URL_PREFIX = ''
      AUTH_REDIRECT_URI = 'urn:ietf:wg:oauth:2.0:oob'
      MASS_TRANSIT_LIVE_KEEP_DAYS = 7
      GMAIL_FROM = 'gmail-from-username@gmail.com'
      GMAIL_PWD = 'password-for-the-above'
      EMAIL_TO = 'email-to-recipient@somewhere.com'
      REVERSE_GEOCODING_URI_TEMPLATE = 'https://api.digitransit.fi/geocoding/v1/reverse?sources=osm&size=20&point.lat={lat}&point.lon={lon}'
      REVERSE_GEOCODING_QUERIES_PER_SECOND = 6
   

   Some explanations:

   • SQLALCHEMY_DATABASE_URI: qwerty is the password for the regularroutes role created here.
   • On the Google Developers Console, create a project (if not already created) and enable Google Maps JavaScript API as explained in devops readme.
   • Still on the Console, create the Browser API key also according to devops readme. Under Accept requests from these HTTP referrers (web sites) enter http://localhost:5000. Press Save
   • From the generated Browser API key, copy the API key value into MAPS_API_KEY in your regularroutes.cfg file shown above.
   • FIREBASE_KEY is found from Firebase console Settings -> Project Settings -> Cloud messaging -> Project Credentials -> Server key. Note that they google-services.json file from the console is needed for the corresponding client.
   • MASS_TRANSIT_LIVE_KEEP_DAYS is the number of days vehicle live data will be stored. Recognised public transportation trips are stored indefinitely. A value of 1 is enough.
   • The current participation cancellation function (in siteserver.py) sends an email with the user_id to the configured EMAIL_TO address. The 'yagmail' library uses the gmail server, so a gmail account is needed (GMAIL_FROM and GMAIL_PWD) for sending.
   • REVERSE_GEOCODING_URI_TEMPLATE is the URI of a Pelias instance for reverse geocoding in regularroutes-site.

   Note: When creating a new server using chef as instructed in devops, the regularroutes.cfg file is automatically generated using parameters from a regularroutes-srvr.json file.

1. Save the client_secrets.json file for your project to the root of your repo. Instructions for generating it are in the devops readme. This file is required in server startup and used in mobile client authentication, but to access this local dev environment from a mobile client also a web server and a routable IP address for the local machine are needed, not covered by these setup instructions.
2. Optional: Install virtualenv, which in addition to pip should be the only global python packages. Can be installed e.g. via pip, easy_install, apt-get. E.g. pip install virtualenv. (Note: Not needed if using the virtualenv built into PyCharm.)
3. Install and run PyCharm IDE for web server / flask development (Note: There is an educational license available for Intellij-Idea and PyCharm ultimate versions.)
4. File Open your regular-routes-repository root folder. If offered, deny package installations from requirements.txt at this point so they don't install into the global environment.
5. Create a virtualenv from: PyCharm/ Preferences / Project: regular-routes-server / Project Interpreter (on Mac, File / Settings / ... in Linux). Click the wheel on the upper right, select Create virtualenv and create a virtualenv named e.g. regular-routes-server with Python v. 3.x.y. Select your virtualenv from the drop-down box next to the wheel. (MJR Note: Currently using 3.6)
6. If the packages in requirements.txt don't start installing otherwise, open one of the root-directory python-files on the editor (e.g. apiserver.py)
7. Install Flask-classy (REST extension package; found from python repo & PyCharm package download list). Under PyCharm installations are done under Preferences (or Settings) / Project / Project Interpreter and currently appear to be included with the packages in the requirements.txt.
8. Install any other packages, which might be missing from requirements.txt.
9. Under Run Edit configurations set the working directory to point to your repository root. Also check the Script:, which can be siteserver.py, devserver.py, apiserver.py or scheduler.py depending on which component is to be run.
10. Run 'regular-routes-server'
11. If no major problems occurred in startup, open a browser window and test. E.g. http://localhost:5000 from siteserver should show the login page.

Python virtualenv

Basically virtualenv and pip should be the only global python packages. Virtualenv isolates the site-packages for each separate project, making it easier to maintain them. For the local development environment virtual environments can be managed with PyCharm, on the server the deployment chef-script creates the necessary virtual environment.

If there is a need to access virtualenvs manually, the installation can be done with pip:

$ sudo pip install virtualenv

Current default setup in the regularroutes servers is that the virtualenv is in directory /opt/regularroutes and it is called virtualenv. Activation from command line:

$ source virtualenv/bin/activate

Sometimes it is necessary to regenerate the virtualenv for cleanup purposes. This can be done with:

1. Rename the old virtualenv to some other name
2. Create a new virtualenv: $ virtualenv virtualenv
3. Install the requirements: pip install -r requirements.txt