With the ever increasing proliferation of IoT devices, and the resulting frequency of data emanating from many devices, the acquisition and management of this data is proving to be a challenge.
In dealing with this challenge, cloud-based sytems that provide service elasticity are helpful in handling the potentially intermittent bursts of high volume data from these devices.
Cristata is a system, targeted at the developer community, to support general IoT data ingestion. It utilises several existing IBM Cloud services and details how these services are wired together. Serverless computing is employed to provide service elasticity and reduce the infrastructure management burden. Sample code includes SQL code for data storage (schema), Openwhisk actions (Cloud Functions) for interfacing with this schema and IoT data submission code.
- An IBM Cloud account is already created (https://console.bluemix.net/)
- With an organisation and space created
- IBM Cloud (Bluemix) CLI (command line interface) is installed (see Vagrantfile for details)
- IBM Cloud (Bluemix) Functions CLI (command line interface) is installed (see Vagrantfile for details)
- Optionally a virtual machine is provisioned.
For ease of development and in particular, deployment to the IBM Cloud, a virtual machine is used which contains the required software and tools. A Vagrantfile is provided, which automates the provisioning of this virtual machine. To use this, VirtualBox and vagrant must be installed. Then it's simply a matter of running:
vagrant up vagrant ssh
Alternatively, the required tools can be installed directly on the host machine. Please refer to the Vagrantfile for dependency information.
IBM Cloud Login
Login, and set organisation and space, using a combination of the commands below:
bx login -a https://api.ng.bluemix.net #or https://api.eu-gb.bluemix.net, https://api.eu-de.bluemix.net bx iam orgs # Outputs available organisations bx iam spaces # Outputs available spaces bx target --cf # Interactively select organisation/space bx target -o <ORG> # Manually select organisation bx target -s <SPACE> # Manually select space
Bulding Helper Components
Cristata is backed by a DB2 schema, which must be deployed. A helper component is built for this purpose using Maven. First, fetch the
IBM Data Server Driver for JDBC and SQLJ (JCC Driver) version 11.1 from here http://www-01.ibm.com/support/docview.wss?uid=swg21363866. This driver (
db2jcc4.jar) must be installed in the local Maven repository before building the helper component.
cp /path/to/db2jcc4.jar ./sql cd sql mvn install:install-file -Dfile=db2jcc4.jar -DgroupId=com.ibm.db2.jcc -DartifactId=db2jcc4 -Dversion=11.1 -Dpackaging=jar mvn clean package cd -
Further details about this component can be found here.
Provision IBM Cloud Components
Cristata's data persistence layer requires an internet-facing DB2 instance as well as Watson IoT and Message Hub. A script to automatically provision and configure these instances is provided. Note: if not bringing up a virtual machine using the Vagrantfile, please check for required packages (openjdk-8-jdk, maven, python3.6, python3-pip, jq etc).
As part of this provisioning process:
- DB2 tables and stored procedures will be created.
- Watson IoT device types and devices will be created.
- Device publish credentials will be created.
- A Message Hub topic will be created.
Note: some of these IBM Cloud components are not free.
Connect Watson IoT to MessageHub
Now, a manual step is required unfortunately, to connect Watson IoT and Message Hub. This is achieved in the Watson IoT dashboard See here for details.
Set up IBM Cloud Functions
- Change into the
openwhiskdirectory, which now contains
my_setup.sh. This was generated automatically by the provisioning process.
- Invoke the makefile to build the actions for IBM Cloud Functions.
- Deploy the actions by invoking the install script.
The commands are:
cd openwhisk make all ./install.sh <BLUEMIX SPACE> cd -
Note: It may take a few moments for the installed IBM Cloud Functions triggers and actions to become ready for use.
Test pipeline by sending a message to Watson IoT
At this point, all components are provisioned and configured. During the provisioning process, credentials for new devices were created (
mqtt-config1/2.json). These credentials can now be used to send data to the platform.
cd iot pip3 install ibmiotf python3 publish.py mqtt-config1.json cd -
To subsequently view all devices added to the Cristata database run:
cd openwhisk bx wsk action invoke cristata-DeviceListing --result
And to view measurements for a given device (as specified in
bx wsk action invoke cristata-TimeSeriesRetrieve -P request.json --result
Action REST API
Actions can also be invoked through their default REST API. The
invoke.sh script provides an example using curl.
Cristata is a sub-component of the GOFLEX H2020 project.
The project Generalized Operational FLEXibility for Integrating Renewables in the Distribution Grid (GOFLEX) has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 731232.