Telco Traffic Simulator
=======================

This application simulates mobile telco traffic.

The main purpose is testing complex event processors, for example built in Spark Streaming.

It can publish simultaneously to multiple brokers, including kafka, sockets, websockets, and logging.
The metrics generated by the celltowers are configurable, and are sampled from user defined distributions.
Metrics can be configured to emit anomalies.

Vocabulary
----------
- 'traffic database' is an sqlite database residing in folder dist/. It contains celltower and subscriber information. See below.
- 'mcc' is the mobile country code, see https://en.wikipedia.org/wiki/Mobile_country_code
- 'mnc' is the mobile network code, see https://en.wikipedia.org/wiki/Mobile_country_code
- 'subscriber' is a random record taken from the traffic database table 'identities'
- 'route' is a route calculated by Google, between 2 random points within the area defined by 'mcc' and 'mnc'
- 'velocity' is the speed at which the subscribers travel, in kilometers per hour
- 'trip' is a combination of a 'subscriber', a 'route', and a 'velocity'
- 'slide' is the size of the sliding publishing window, in milliseconds. That is: how often do you want the events published.
- 'broker': where to publish the events ? See conf/application.conf.

How does it work ?
------------------
Simulations are invoked by sending a simulation request to the service.
You can launch a simulation through REST or through the web interface.
By default the web interface is accessible at http://localhost:9000/simulator.
See here for a screen shot: file://doc/screenshot.png
Example scripts for using the REST interface are available in the bin/ folder.

A simulation request consists of the following parameters:
- mcc
- mnc
- number of simultaneous trips
- slide size
- velocity

For each trip, a random subscriber and 2 random celltowers are picked from the database.
A request is sent to Google to calculate the route between the locations of the 2 celltowers.
The subscriber is then sent on its way along this route, at the speed defined by the velocity.

Every 'slide' milliseconds, the location and the details of the subscriber will be published to the 'subscriber-topic' of the configured broker.
At the same time, the celltower nearest to the subscriber is retrieved.
This celltower publishes its details along with a bearer id and metrics to the 'celltower-topic'.
The bearer id allows to link the subscriber to the celltower event.
The metrics are configurable in file conf/celltowers.conf.
Every metric template consists of a metric name, a mean and a standard deviation.
When a celltower event is published, a sample is taken from the gaussian distribution defined by these parameters.

Note that currently all events are published in JSON format.

Creation of traffic.db
----------------------
Before you can use the application, you need the traffic database.

To create the traffic.db database, you will need an installation of sqlite3 (see https://www.sqlite.org/ )
You will also need an API key from http://opencellid.org/
You also need wget (https://www.gnu.org/software/wget/) if you want to follow the instructions below

First, get the celltower database from opencellid.org, by executing:

bin/download_celltower_db.sh OPENCELLID_API_KEY

Note: if you have a free account with opencellid, you will only be able to do this once a day.

Now create the database, by executing:

bin/create_traffic_db.sh

Running the simulator
---------------------

You will need a Google API key from https://developers.google.com/maps/documentation/javascript/get-api-key.
Once you have they you must create a file conf/cred.conf and enter it in there:
google.api.key=YOUR_API_KEY

Before starting the application, please verify the settings in conf/application.conf.

To start the application, execute
./activator run
in the root of the checked out code.
If you do this for the first time, then it might take a while before all dependencies are downloaded.
By default this will start the web server on localhost:9000 and will use port 2552 for akka communication.

To start more nodes, and run the application as a cluster, execute
./activator run -Dhttp.port=N -Dakka.remote.netty.tcp.port=0
(Replace N by a unique number. You could also use 0 but in development mode you need to refresh the web browser for the application to become active.)

For detailed cluster configuration, see the akka documentation at
http://doc.akka.io/docs/akka/2.4.1/scala/cluster-usage.html
and the file
conf/application.conf

Building a production distribution
----------------------------------

Prerequisites: the traffic database has been created
In the root of the checked out code, execute: 

./activator clean universal:packageZipTarball

After a while, the file target/universal/play-traffic-1.0-SNAPSHOT.tgz wil have been created.
You can copy this file to your target platform, and extract it in a working folder:

tar xvfz play-traffic-1.0-SNAPSHOT.tgz

Execute the following commands to get the server started
cd play-traffic-1.0-SNAPSHOT
bin/play-traffic -Dhttp.port=9000 -Dakka.remote.netty.tcp.port=2552

To deploy a cluster, update conf/application.conf if necessary, and execute new nodes as follows:
bin/play-traffic -Dhttp.port=0 -Dakka.remote.netty.tcp.port=0

Event model
-----------

The event model used by the application really is very simple, and bears almost no resemblance to real telco network traffic.
Nevertheless, it can be used for testing use cases like geofencing and anomaly detection.

Events are emitted to the brokers, and are currently only available in JSON format.
There are 3 types of events:
- subscriber events
- celltower events
- attach events

Subscriber events are events that serve for visualisation only by this application (if you have the websocket broker enabled).
They are published to the subscriber-topic of the broker.
Location of a subscriber should not be derived from these events, since the exact location is part of the event.
Example:
{
  "subscriber": {
    "id": 2167,
    "imsi": "5294787564",
    "msisdn": "475012128",
    "imei": "968276283796837",
    "lastName": "Vlastuin",
    "firstName": "Jorgen",
    "address": "Blekersdijk 235",
    "city": "Dolembreux",
    "zip": "4140",
    "country": "Belgium"
  },
  "location": {
    "lat": 50.38687346229368,
    "lng": 5.68108974039459
  },
  "bearerId": "c3906369-c6c1-43c6-bc46-78143c23edee",
  "topic": "subscriber-topic"
}

Celltower events are defined in conf/celltowers.conf.
They are published to the subscriber-topic of the broker.

The celltower event definisiotn looks like this:

"celltower-templates" = [

  {
    name: "celltower-template-1" # template name
    metrics: [
      {
        name: "rtt"  # metric name
        mean: 5.0 # mean of the normal distribution for this metric
        std: 1.2  # standard deviation of the normal distribution for this metric
        anomalyFrequency: 10 # 1 in 10 messages will be an anomaly
      }
      {
        name: "a12b5"
        mean: 7.0
        std: 0.5
      }
    ]
  }
  ...
]

This file contains a number of templates, which are randomly chosen when a celltower becomes active.
The template stays with the celltower for the duration of a simulation.

The metrics section contains any number of metric definitions.
Each metric definition consists of a name, a mean and a standard deviation, and an optional anomalyFrequency.
The mean and the standard deviation are used to build a gaussian distribution, which is used for sampling.
The anomalyFrequency is an integer that defines the number of anomalies that will be emitted.
For example anomalyFrequency 10 means 1 in 10 messages will be an anomaly.
An anomaly if taken from the gaussian distribution where the mean = the original mean + 2 * the standard deviation.
The standard deviation is the original one.

Example of a celltower event:
{
  "celltower": {
    "mcc": 206,
    "mnc": 20,
    "cell": 3368384,
    "area": 262,
    "location": {
      "lat": 50.812583,
      "lng": 5.569499
    }
  },
  "bearerId": "195cac1c-97e1-4388-a097-ecd88684e5c5",
  "metrics": {
    "rtt": 103.65686007107024,
    "a12b5": -0.1472962620330414,
    "eventCounter": 2
  },
  "topic": "celltower-topic"
}

Attach events are only emitted once, when the subscriber start the trip.
Attach events are published to the attach-topic of the broker.
This event contains the bearerId, which allows the celltower events to link to the subscriber.
(There is no subscriber information in a celltower event).
Example of an attach event:
{
  "bearerId": "64f22a7d-f782-4dd1-af55-a589c7e22bde",
  "subscriber": {
    "id": 826,
    "imsi": "6526989658",
    "msisdn": "475057422",
    "imei": "153899429961484",
    "lastName": "Zimmerman",
    "firstName": "Ines",
    "address": "Putstraat 435",
    "city": "Slijpe",
    "zip": "8433",
    "country": "Belgium"
  },
  "topic": "attach-topic"
}

TODO
----
see https://github.com/koen-dejonghe/play-traffic/issues

If you have ideas, please send them to
koen@fietsoverland.com

Thank you.