Skip to content

Real time Streaming of Usage Data

Martin Giess edited this page Aug 24, 2017 · 22 revisions

This page is outdated, pls have a look at Real-time Streaming of Event and Usage Data


The EMnify Data Streamer (EDS) allows you to receive real-time data streams with usage data of your endpoints and SIMs on your server or to push the data to one of the pre-integrated cloud services such as keen.io, DataDog, AWS Kinesis or AWS S3.

EDS Overview

You can create up to 3 data streams.

Configuration of EDS

Please log in to your EUI account and click on the "link" icon on the right/top corner, from there you have access to the data streams panel.

EDS Add Data Stream

To configure a new data stream please perform the following actions:

  • click on the action button "Add Data Stream"
  • choose the API type, which can be
    • EMnify API
    • keen.io
    • DataDog
    • AWS Kinesis
    • AWS S3
  • depending on the API type further API keys and configuration parameters may be required (see below for configuration of the different integrations)
  • check "Stream historic data" if you want to have past data send to your server, in this case the system will send you data up to 20 days old and it may take some time to catch up with live data
  • click "Add Data Stream"

EDS Data Streams Panel

The new data stream will instantly become active and you can monitor its status on the info panel, the column "last result status" shows you the HTTP status code from the remote site. This should be 200 (OK) under normal condition, if the receiving part is down or the configuration is valid it will show 5XX errors.

At any time you can pause a stream (e.g. if you want to take down the receiving server for maintenance) and resume once convenient by using the action buttons right to the stream info.

If want to remove a data stream permanently please use the remove icon on the right, you will be asked to confirm this action.

You can create up to 3 data stream that will run in parallel, e.g. you can have a stream going to your monitoring system or data analytics platform and one to S3 for archiving at the same time.

EDS native JSON format

If you choose to implement the EMnify API on your server then EDS will post data with the following JSON object:

{
"cost": 0.00558275,
"id": 174321498,
"operator": {
		"id": 4,
	"name": "EPlus",
	"country": {
		"id": 74,
		"name": "Germany"
	}
},
"organisation": {
	"id": 399921,
	"name": "Test"
},
"tariff": {
	"ratezone": {
		"id": 83,
		"name": "Europe_I"
	},
	"id": 64,
	"name": "Global Pro I"
},
"traffic_type": {
	"id": 5,
	"name": "Data"
},
"endpoint": {
	"id": 8392037
},
"imsi": "901430999910777",
"volume": {
	"rx": 0.0138,
	"tx": 0.008531,
	"total": 0.022331
},
"start_timestamp": "2017-03-19 21:06:33",
"sim": {
	"iccid": "8988303000000011085",
	"id": 233746
},
"currency": {
	"symbol": "€",
	"code": "EUR",
	"id": 1
},
"end_timestamp": "2017-03-19 21:21:23"
}

Integrations with external services

keen.io

keen.io offers APIs for streaming, analyzing, and embedding rich data (http://www.keen.io) and integrates nicely with EDS.

You can get EDS running with keen.io in minutes to run your first queries and build dashboards, following steps are required:

  • Create a keen.io user account
  • Create a new project, e.g. "EMnify Usage Data Stream"
  • In the project settings go to "Access" to find your project id and write key EDS keen.io access data
  • Log on to the EUI and add a keen.io stream EDS keen.io add stream

Shortly after the stream is created you will see first data arriving, which you can explore on keen.io on the "streams" tab. There choose data collection "EMnifyUsageData" and click on "Last 10 Events Streamed".

EDS keen.io add stream

DataDog

DataDog provides real-time performance monitoring (https://www.datadoghq.com). In conjunction with EDS it allows you to collect and analyse metrics about the usage of your endpoints and SIM cards, you can create dashboards and trigger alerts on certain situations.

  • Create a DataDog user account
  • Go to "Integrations" and "APIs" from the left navbar and generate a new Application Key
  • Log on to the EUI and add a DataDog stream EDS DataDog add stream

Shortly after the stream is created you will see first data arriving, in the DataDog explorer you can check the incoming data and then you can start to create dashboards.

Following metrics are provided

  • endpoint.volume
  • endpoint.volume_tx
  • endpoint.volume_rx
  • endpoint.cost

AWS Kinesis

  • Log on to your AWS account
  • Switch to region "eu-west-1"
  • Go to "Kinesis" and create a new stream, call it "EMnifyUsageData"
  • Go to "IAM" and create a new user limited to Kinesis server and that stream
    • you will need to create and store the Access Key and Secret Key
  • Log on to EUI and add a Kinesis Data Stream EDS Kinesis add stream

Currently only region "eu-west-1" is supported by EDS.

AWS S3

  • Log on to your AWS account
  • Go to "S3" and create a new bucket
  • Go to "IAM" and create a new user limited to that S3 bucket
    • you will need to create and store the Access Key and Secret Key
  • Log on to EUI and add a S3 Data Stream EDS S3 add stream

Shortly after the stream is created you will see CSV files arriving in your S3 bucket.

EDS S3 bucket

You can’t perform that action at this time.