Detect anomalies in data streamed to "Maximo Asset Monitor"

In this Code Pattern we will show how to publish data stored within an Watson IOT Platform Analytics instance to an external service. Once the data is analyzed by the service, the results can then be persisted in the Analytics Platform database.

Our specific demonstrated use case here is to detect anomalies within a Maximo Asset Monitor dataset using a scikit-learn model, which is externally hosted in a Watson Machine Learning service. We'll also demonstrate how to visualize correlations between anomalies via time-series graphs.

When the reader has completed this Code Pattern, they will understand how to:

• Load asset data into Watson IOT Platform Analytics.
• Forward data to external services via REST HTTP call.
• Build a dashboard using Maximo Asset Monitor to monitor, visualize, and analyze IOT asset data.
• Generate alerts when certain results are received.

The intended audience for this Code Pattern are developers / data scientists who would like to analyze their data in Watson IOT Platform Analytics via customized machine learning models that are hosted externally.

Flow

1. IoT data is retrieved by function in Watson IOT Platform Analytics
2. Data is packaged as a dataframe and forwarded to a REST endpoint where custom model is hosted
3. Model results are retrieved and appended to dataset
4. Dataset is validated and committed to DB, alert is sent if anomaly is detected

Components

• Watson IOT Platform Analytics. This is a SaaS offering that allows you to register devices, collect IOT Data and build IOT applications. This add-on service extends "Watson IoT Platform" to include Maximo Asset Monitor. Sign up for an account here

• HTTPPreload This is a prebuilt analytics function that allows you to collect IOT asset and sensor data from external data sources.

• Watson Machine Learning This is a service that enables developers to expose their machine learning models via a REST interface.

Prerequisites

• An account on IBM Marketplace that has access to Watson IOT Platform Analytics and Maximo Asset Monitor. This service can be provisioned here

• Python 3, pip

Steps

Follow these steps to setup and run this Code Pattern.

1. Provision cloud services
2. Setup your Python development environment
3. Leverage Python scripts to register entity, function, and ML model
4. Deploy Function
5. Add alerts
6. Add Dashboard Visualizations

1. Provision Cloud Services

Navigate to the IBM Cloud dashboard at https://cloud.ibm.com/ and click the "Catalog" button in the upper right

Search for the "Watson Machine Learning" service and select the icon

Select the pricing plan and click "Create". If deploying on an IBM Lite account, be sure to select the free "Lite" plan

Navigate back to the IBM Cloud catalog https://cloud.ibm.com/catalog

Search for the "Watson Studio" service and select the resulting icon

Select the pricing plan and click "Create". If deploying on an IBM Lite account, be sure to select the free "Lite" plan

Once the service has been provisioned, select the "Get Started" button to enter the Watson studio console

Once the Watson Studio dashboard has loaded, scroll to the "Projects" section and click "New Projects" to create a project.

Select a "Empty Project" as the project type

Enter a project name, and then click "Create"

Once the project has been created, a dashboard showing all the project assets will be displayed.

Finally, we'll need to associate our "Watson Machine Learning" service with our project. We can do so by clicking "Settings"

Then, scroll down to the Associated services section. Click "Add Service" -> "Watson".

Click "Machine Learning"

Find your instance on the drop-down list, and then click "Select"

After creating the Watson Machine Learning and Watson Studio instances, we'll then need to create a configuration file containing the authentication/project details.

First, take note of the url of your Watson Studio instance. After the /projects endpoint, you'll see a UUID for that particular project.

Place the project uuid in a file titled wml_credentials.json like so

{
  "project_id": "<project_uid>"
}

Then generate an IBM Cloud API key (if you don't have one already). Begin by visiting the following url https://cloud.ibm.com/iam/apikeys

Then click the blue button titled "Create an IBM Cloud API key"

Give the API key a name and click Create

Copy the API key and place it in the configuration file. The final wml_credentials.json file should then look like so

{
 "url": "https://us-south.ml.cloud.ibm.com",
 "project_id": "<project_uid>",
 "apikey":"***********"
}

2. Setup your Python development environment

Install Python

Clone this repository

git clone https://github.com/IBM/iot-analytics-anomaly
cd iot-analytics-anomaly

Instal virtualenv (if you don't have it already)

sudo pip install virtualenv

Create a virtual environment

python3 -m venv env

Activate Virtual Environment, Install Python Dependencies and Verify Environment

Enter your Virtual Environment directory

cd env

Activate your virtual environment

source bin/activate

Install dependencies

# Prereqs
pip install numpy
pip install sqlalchemy pandas ibm_db_sa urllib3 requests lxml sklearn ibm_db python-dotenv future

# Watson IOT Functions
pip install git+https://github.com/ibm-watson-iot/functions.git@production --upgrade
pip install -r requirements.txt

# Watson Machine Learning Client
pip install watson_machine_learning_client

3. Leverage Python scripts to register entity, function, and ML model

Copy your Watson IOT Platform Service credentials into a credentials.json file

Navigate to your Watson IOT Platform Analytics service. This should be accessible at a url like so

https://dashboard-us.connectedproducts.internetofthings.ibmcloud.com/preauth?tenantid={tenant-id}

Expand the left menu,

Click "Services" -> "Watson IOT Platform Analytics" -> "View Details" -> "Copy to clipboard"

If you've created a custom fork of this repo, modify your .custom/functions.py to set your PACKAGE_URL as the forked Github repository. This is necessary because the Analytics Service will pip install the custom function from that url.

PACKAGE_URL = 'git+https://github.com/ibm/IBM/iot-analytics-anomaly'

Change the class name if someone else has already published a function with the same name in your tenant function catalog. In this case, our default function name is InvokeModel.

If there is already a function named as "InvokeModel", you'll need to change the function name

For example, if you'd like your function name to be "CustomFunctionName", change the class name in custom/functions.py

class CustomFunctionName(BaseTransformer):

Next, set the PYTHONPATH to the root directory of this project

export PYTHONPATH=$(pwd)

Finally, create a .env file using the template in custom/template.env as a guide. The next step will require the following values to be set. Be sure to replace the example values with the values that correspond to your entity type.

# Entity Values
ENTITY_NAME="turbine_demo"
ENTITY_IDS="73000,73001"
INPUT_COLUMNS="drvn_flow,drvn_t1,drvn_t2,drvn_p1,drvn_p2"

# Credentials
MONITOR_CREDENTIALS_PATH="credentials/monitor-credentials.json"
WML_CREDENTIALS_PATH="credentials/wml-credentials.json"

Run Notebook

Now we can begin running the generate_WML_model.ipynb notebook. This notebook leverages the iotfunctions library to interact with the analytics service.

Run the following command

juypter-notebook

And access the notebook at http://localhost:8888/notebooks/generate_WML_model.ipynb

In this example, we'll use the notebook to build a "IsolationForest" anomaly detection model, which is included with the scikit-learn python package. This model will read features from each row, and determine whether the row data is anomalous or not. A value of "-1" or "1" will be placed in the "anomaly_score" column in each row after the function runs.

• -1 if data in a row seems to be an outlier (anomaly detected).
• 1 if the row data seems to be an inlier (normal).

As the notebook runs, check the output of each cell to confirm no errors are being thrown.

In the final cell, we can test the registered function.

4. Deploy Function

Next, we'll add our custom function to our newly created entity via the UI. This will enable the selected function to run every 5 minutes and analyze new data for anomalies. To enable the function, navigate to the "Add Data view". We can get to this form in the Analytics dashboard by selecting the following

Monitor > Entity Types > (entity name) > Data

Click the + button and search for "InvokeModel"

Enter the values/credentials for your Machine Learning instance. You can also specify which features to select.

• URL - WML Endpoint
• apikey - "WML API key"
• model_id - "WML Model ID (should be printed after running invoke_model_function.py)"
• deployment_id - "WML Deployment ID (should be printed after running invoke_model_function.py)"
• input_features - "comma separated list of features that should be loaded from entity dataframe"

Then enter an output variable name and click "Create". The output variable is a boolean value indicating whether the function completed successfully.

In summary, the registered function will check for new data from the associated entity instances every five minutes. The relevant columns (input_features) in the newly added data is then forwarded to the external ML model, and the result is stored in the "anomaly_results" column of the dataframe.

5. Add alerts

We can also Leverage the "Alerts" feature to receive notifications as anomalies are detected. We'll do so here by clicking the + button in the Entity view, and then selecting the "AlertExpression" function

Then, enter a condition indicating when the Alert should be triggered. In this case, we should get an alert whenever our anomaly score value is below 0. Enter an output value (using "anomaly_detected" here) and click next.

We can then see a list of all anomalies by selecting our "anomaly_detected" variable.

6. Add Dashboard Visualizations

We can also generate visualizations for each entity instance, which can help identify high level patterns in a dataset. This is done by selecting one of the records in the "instance dashboards" table

Next, click the gear icon in the upper right, and select "Edit Dashboard"

Click "Import"

Select the json file in dashboards/speed_line_dashboard.json

This will show an overview of instance data for any particular entity.

Metrics can be added to the chart by adding objects to the content and dataSource arrays in the dashboards/anomalyInstanceDashboard.json file

{
    "dataSourceId": "<metric_name>",
    "label": "<metric_name>"
}

...

{
    "aggregator": "max",
    "attribute": "<metric_name>",
    "id": "<metric_name>"
},

Learn more

• Watson IOT Platform Code Patterns: Enjoyed this Code Pattern? Check out our other Watson IOT Platform Code Patterns.

• Knowledge Center:Understand how this Python function can load data into Watson IOT Platform Analytics

License

This code pattern is licensed under the Apache Software License, Version 2. Separate third party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 (DCO) and the Apache Software License, Version 2.

Apache Software License (ASL) FAQ