Azure IoT Hub Streaming, Analytics, Alerts generation
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.github Sync project with template and rename project (#16) Jul 15, 2017
app
conf
project
scripts
test
.gitattributes
.gitignore
.travis.yml
CONTRIBUTING.md
LICENSE
README.md
build.sbt
intellij-code-style.xml
sbt
sbt.cmd

README.md

Build Issues Gitter

Telemetry Agent Overview

This service analyzes the telemetry stream, stores messages from Azure IoT Hub to DocumentDb, and generates alerts according to defined rules. The IoT stream is analyzed using a set of rules defined in the Telemetry service, and generating "alarms" when a message matches some of these rules. The alarms are also stored in DocumentDb.

We provide also a .NET version of this project.

Dependencies

How to use the microservice

Quickstart - Running the service with Docker

  1. Install Docker Compose: https://docs.docker.com/compose/install
  2. Create an instance of an Azure IoT Hub
  3. Store the "IoT Hub" connection string in the env-vars-setup script. The service internally uses IoT Hub React so you will notice some variables with an "IOTHUBREACT" prefix. For more information about environment variables, see the Running the service with IntelliJ IDEA.
  4. Open a terminal console into the project folder, and run these commands to start the Telemetry Agent service
    cd scripts
    env-vars-setup      // (Bash users: ./env-vars-setup).  This script creates your env. variables
    cd docker
    docker-compose up
    

The Docker compose configuration requires the dependencies resolved and environment variables set as described previously. You should now start seeing the stream content in the console.

Java setup

  1. Install the latest Java SDK.
  2. Use your preferred IDE, IntelliJ IDEA and Eclipse are the most popular, however anything should be just fine.

Running the service with IntelliJ IDEA

  1. Install Docker: https://docs.docker.com/engine/installation
  2. Create an instance of Azure IoT Hub
  3. Install IntelliJ IDEA Community 2017, with SBT plugin enabled
  4. Open the solution in IntelliJ IDEA
  5. "Open" the project with IntelliJ, the IDE should automatically recognize the SBT structure. Wait for the IDE to download some dependencies (see IntelliJ status bar)
  6. Create a new Run Configuration, of type "SBT Task" and enter "run 9023" (including the double quotes). This ensures that the service will start using the TCP port 9023
  7. Either in IntelliJ Run Configuration or in your system, define the following environment variables:
    1. PCS_TELEMETRYAGENT_DOCUMENTDB_CONNSTRING = {your CosmosDb DocumentDb connection string}
    2. PCS_TELEMETRY_WEBSERVICE_URL = {the Telemetry service endpoint}
    3. PCS_CONFIG_WEBSERVICE_URL = {the Config service endpoint}
    4. PCS_IOTHUBMANAGER_WEBSERVICE_URL = {the IoT HubManager service endpoint}
    5. PCS_IOTHUBREACT_AZUREBLOB_ACCOUNT = {your Azure Blob Storage account name}
    6. PCS_IOTHUBREACT_AZUREBLOB_KEY = {your Azure Blob Storage account key}
    7. PCS_IOTHUBREACT_HUB_NAME = {your Azure IoT Hub - Event Hub compatible name}
    8. PCS_IOTHUBREACT_HUB_ENDPOINT = {your Azure IoT Hub endpoint}
    9. PCS_IOTHUBREACT_HUB_PARTITIONS = {your Azure IoT Hub partition count}
    10. PCS_IOTHUBREACT_ACCESS_CONNSTRING = {your Azure IoT Hub access connection string}
  8. Start the StreamingAgent project
  9. Using an HTTP client like Postman, test if the service is up and running http://127.0.0.1:9023/v1/status

You should now start seeing the stream content in the IDE console.

Running the service with Eclipse

The integration with Eclipse requires the sbteclipse plugin, already included, and an initial setup via command line (more info here).

Steps using Eclipse Oxygen ("Eclipse for Java Developers" package):

  1. Open a console (either Bash or Windows CMD), move into the project folder, execute sbt compile and then sbt eclipse. This generates some files required by Eclipse to recognize the project.
  2. Open Eclipse, and from the Welcome screen "Import" an existing project, navigating to the root folder of the project.
  3. Run the env-vars-setup script after adding the requried values
  4. From the console run sbt -jvm-debug 9999 "run 9023" to start the project
  5. Test that the service is up and running pointing your browser to http://127.0.0.1:9023/v1/status
  6. In Eclipse, select "Run -> Debug Configurations" and add a "Remote Java Application", using "localhost" and port "9999".
  7. After saving this configuration, you can click "Debug" to connect to the running application.

Project Structure

This microservice contains the following projects:

  • Code for the application is in app/com.microsoft.azure.iotsolutions.telemetryagent/
    • WebService - Java web service exposing REST interface for status checkpoint to verify that the dependencies are resolved
    • Services - Java project containing business logic for interacting with Azure services (IoTHub, DocDb etc.) and device telemetry microservice
    • StreamingAgent - Java project that reads messages from IoT Hub, processes them using rules that generate alarms and store alarms and messages in the storage
  • Tests
    • WebService - Unit tests for web services functionality
    • Services - Unit tests for services functionality
  • Scripts - contains build scripts, docker container creation scripts, and scripts for running the microservice from the command line
  • Routes - defines the URL mapping to web service classes

Build & Run from the command line

The scripts folder includes some scripts for frequent tasks you might want to run from the command line:

  • compile: compile the project.
  • build: compile the project and run the tests.
  • run: compile the project and run the service.

The scripts check for the environment variables setup. You can set the environment variables globally in your OS, or use the env-vars-setup in the scripts folder.

If you are familiar with SBT, you can also use SBT directly. A copy of SBT is included in the root of the project.

Updating the Docker image

The scripts folder includes a docker subfolder with the files required to package the service into a Docker image:

  • build: build a Docker container and store the image in the local registry
  • run: run the Docker container from the image stored in the local registry script

You might notice that there is no Dockerfile. All Docker settings are defined in build.sbt.

dockerRepository := Some("azureiotpcs")
dockerAlias := DockerAlias(dockerRepository.value, None, packageName.value + "-java", Some((version in Docker).value))
dockerBaseImage := "toketi/openjdk-8-jre-alpine-bash"
dockerUpdateLatest := false
dockerBuildOptions ++= Seq("--compress", "--label", "Tags=Azure,IoT,PCS,Java")
dockerEntrypoint := Seq("bin/telemetry-agent")

The package logic is executed via sbt-native-packager, installed in plugins.sbt.

You can also start Telemetry service and its dependencies in one simple step, using Docker Compose with the docker-compose.yml file in the project:

cd scripts/docker
docker-compose up

The Docker compose configuration requires the dependencies resolved and environment variables set as described previously.

Configuration and Environment variables

The service configuration is stored using Akka's HOCON format in application.conf.

The HOCON format is a human readable format, very close to JSON, with some useful features:

  • Ability to write comments
  • Support for substitutions, e.g. referencing environment variables
  • Supports JSON notation

The configuration file in the repository references some environment variables that need to created at least once. Depending on your OS and the IDE, there are several ways to manage environment variables:

Contributing to the solution

Please follow our contribution guildelines. We love PRs too.

Troubleshooting

Feedback

Please enter issues, bugs, or suggestions as GitHub Issues here