Data exporter

In the Monroe research project we need to collect data from several different sources (modem metadata, gps, ...) and write it to different outputs (SQLite, ZeroMQ, ...). This required us to write a generic and easily extendible tool.

Data exporter is a basic application for collecting and storing data. It is based around the concept of input and writers, and the application is easy to extend. Everything is table driven and self-contained, so adding new inputs or writers is a matter of adding a new entry to the table(s) and implementing the required callbacks.

An input receives event a source and parses it into a data structure. All the different structures generated by inputs have a common header, and the structures are passed to the writers. The writers then writes the data contained in the structure to the desired output (for the supported structures).

Currently, we support Netlink as input (from an internal tool) and SQLite and ZeroMQ (Monroe) as output. Instead of the earlier mentioned data structure, it could be more optimal to send for example raw JSON to a writer. This would be as easy as adding a new handle callback. The current input/writers is not of much use outside of Monroe, but should serve as nice examples of how to write input/writers.

Pull requests are always welcome! One nice feature would be to parallelize the writers. It should be fairly easy, "all" that is needed is a reference counter on each metadata object passed to the writers (and atomic update of this ref. counter).

Compile instructions

Data exporter is built with CMake and the available writers are selected at compile time. In order to add SQLite, the following flag must be added to CMake:

-DSQLITE3=1

ZeroMQ is added with the following flag:

-DZEROMQ=1

GPSD supports is added with the following flag:

-DGPSD=1

After that, it is just to run make.

Command line options

To get the supported command line options for the core application, run:

./meta_exporter --help

To see the options for the different inputs/writers, they must be specified. For example:

for the core application, run:

./meta_exporter --help

To see the options for the different inputs/writers, they must be specified. For example:

./meta_exporter -s -z --help 

At least one input and one writer must be specified. Here is an example of how to start the application with netlink as input and sqlite as writer:

./meta_exporter -s --sql_database ./abcd.db -t --sql_interval 30 --sql_nodeid
12 --netlink --sql_meta_prefix ./metadata --sql_gps_prefix ./gps --sql_events
200

Metadata description

This part is relevant for those using a Monroe experiment node.

Event types

We currently export four event types:

• 0x01: Interface events. Generated by network listener and is currently only exported for modems. Contains information about mode, signal strength etc.
• 0x02: Connection events. Generated by the Dynamic Load Balancer and contains information about the L3/L4 connectivity.
• 0x03: GPS.
• 0x04: Data usage. Data usage (since last message) for a given interface.

The event type is stored in event_type in the JSON.

Interface events

Interface events have event type 0x01 and will be generated by network listener. We currently only export events for modems.

Each message, until we have selected one identifier, containes the IMSI, ICCID and IMEI. No information is exported until these values are available, i.e., PIN has to be unlocked. See the wiki for complete JSON examples.

Event parameters

A interface event can be one of the following types (stored in event_param) in the JSON object:

• DEVICE_STATE_ CHANGED (1): - sent when device state is changed (see possible states below).
• MODE_ CHANGED (2) - sent when there is a connection state change (for example 3G connection changed to LTE).
• SIGNAL_STRENGTH_CHANGED (3) - sent when signal strength changed.
• LTE_BAND_CHANGED (4) - sent when LTE band is changed (only valid for LTE capable modems).
• ISP_NAME_CHANGED (5) - sent when ISP name (read from modem) has changed.
• META_UPDATE (6) - this event is sent periodically (every 30s) and contains all available informations about modem at the moment.
• IP_ADDRESS_CHANGED (7) - sent when external IP has changed (for example due to end of DHCP lease).
• LOCATION_CHANGED (8) - sent when LAC (Location Area Code), CID (Cell ID), LTE PCI (LTE Physical Cell ID) or EnodeBId (E-UTRAN NodeB ID) has changed.
• NETWORK_MCC/MNC_CHANGED (9) - sent when values of MCC (Mobile Country Code) or MNC (Mobile Network Code) read from network has changed.

Device state

The DEVICE_STATE_CHANGED contains the device state that has been changed to. The values can be the following:

• UNKNOWN (0): Modem state is unknown.
• REGISTERED (1): Modem is registered to the network.
• UNREGISTERED (2): Modem is unregistered from the network.
• CONNECTED (3): Modem is connected to the network.
• DISCONNECTED (4): Modem is disconnected from network.

Device mode/submode

MODE_CHANGED is export when the modem mode/submode changes. The following values for mode are possible:

• UNKNOWN (0).
• DISCONNECTED (1).
• NO_SERVICE (2).
• 2G (3).
• 3G (4).
• LTE (5).

Submode can be one of the following:

• UNKNOWN (0).
• UMTS (1).
• WCDMA (2).
• EVDO (3).
• HSPA (4).
• HSPA+ (5).
• DC HSPA (6).
• DC HSPA+ (7).
• HSDPA (8).
• HSUPA (9).
• HSDPA+HSUPA (10).
• HSDPA+ (11).
• HSDPA+HSUPA (12).
• DC HSDPA+ (13).
• DC HSDPA + HSUPA (14).

Other values

All signal strength values (RSSI, RSRQ, RSRP) are in dBm.

The LTE frequence is in mHz.

We read (and store) the operators in three different fiels. isp_name is the literal ISP name read from modem, while network_mccmnc is the MCC and MNC read from the modem. imsi_mccmnc is the MCC and MNC read from IMSI.

Connection events

Connection events have event type 0x02 and is generated by Dynamic Load Balancer.

Event parameters

A connection event can be one of the following types (stored in event_param) in the JSON object:

• L3_UP (1): Interface has got IP address.
• L3_DOWN (2): Interface has lost IP address.
• L4_UP (3): Interface (address) has internet connectivity.
• L4_DOWN (4): Interface (address) has lost internet connectivity.
• MODE_CHANGE (5): Connection mode has changed (2G, 3G, LTE, etc.).
• QUALITY_CHANGE (6): Quality of connection has changed.
• META_UPDATE (7): Keep-alive message generated every 30th second.
• META_MODE_CHANGE (8): Fake mode change, inserted for two minutes after application has crashed in case proper event has been lost. Only inserted if mode in update message is different than what is stored in DB.
• META_QUALITY_CHANGE (9): Same as fake mode change, but for quality.
• DATA_USAGE_UPDATE (10): Data usage since last update.

Interface type

Each message from Dynamic Load Balancer contains the interface type (stored in interface_type). The following types are defined:

• UNKNOWN (0).
• MODEM (1).
• PHONE (2).
• USB LAN (3).
• WAN (4).
• WIFI (5).

Connection modes

Connection modes can have the following values (stored in event_value):

• UNKNOWN (1).
• DISCONNECTED (2).
• NO SERVICE (3).
• 2G (4).
• 3G (5).
• LTE (6).

Connection qualities

Connection quality can have the following values (stored in event_value):

• UNKNOWN (0).
• BAD (1).
• GOOD (2).

Update message

The update message contains a CSV in the event_value_str field. It is formatted as follows:

w, x, y, z

w is l3 connectivity (0/1), x is l4 connectivity, y is mode and z quality.

GPS events

GPS events have type 0x03 and can come from a multitude of sources. The netlink input expects them to be contained in the variable "nmea_string" in a JSON object. It is possible to timestamp the positions by include a timestamp in the variables "timestamp" in the same object.