This package has been successfully deployed to both a Raspberry Pi and an NXP imx6 modular gateway. Whichever edge you use searches for and connects to any nearby Thunderboards via BLE. The gateway gathers data from the Thunderboards, and then forwards it to the platform. The platform then displays the data in a graph in the AnomalyDetection portal.
This is an ipm package, which contains one or more reusable assets within the ipm Community. The 'package.json' in this repo is a ipm spec's package.json, here, which is a superset of npm's package.json spec, here.
What this system does:
- Flash the firmware of your Thunderboard so the advertisement period for connecting via BLE lasts indefinitely. See the ClearBlade Adapters ThunderBoard Sense Firmware Config pdf for reference.
- Create a system on the ClearBlade platform and install this IPM.
- Go to the Users tab and create a User. Make sure you can remember your password.
- Go to the Adapters tab, select the ThunderBoardAdapter adapter, click the pencil edit icon next to Configuration, and download the pythonScanner.py file. Open it up and change the credentials at the start of the file. Leave the platformURL alone, you can find the System Key and System Secret on the platform if you select the info tab, and then the System Settings subtab. The username and password will be the same as the User you just created. After you've changed the credentials, save the file, go back to the edit configuration modal in the Adapters tab, and replace the python scanner file.
- Go to the Edges tab, click on the gear icon next to the already created edge named ThunderboardOnPi. Select Setup Instructions, and change the target dropdown to Linux 32bit-ARM. Don't close this.
- SSH into your gateway and navigate to the directory you want the edge and adapter software to be in. Go back to your browser, and in the modal you didn't close from the last instrcution, press the copy button next to the Download textbox. Go back to your terminal, paste the instruction, and run it. Do this for Download, Unzip, Install, Permission. You will also need to do this for the command at the very bottom (starts with edge, should have a copy button), BUT you need to put nohup in front of that command so it will log all of it's output in a nohup.out file instead of your terminal.
- Go back to your ClearBlade system, and go to the Adapters tab. Select the ThunderBoardAdapter adapter, select the ThunderboardOnPi edge, and press the play button.
- Congratualtions, you're done! Wait for a minute or two and go to the devices tab. If your gateway found a Thunderboard it should have created a device there. You can also check the Messages tab, if you don't see any MQTT messages with thunderboard in the topic name then something went wrong.
To see a visualization of the data, go to the Portals tab and click on the AnomalyDetection portal. Change the topic field on the right to be thunderboard/environment/<THUNDERBOARD_ID>/_platform (the ID will be a 5 digit number in pretty much every MQQT topic in the Messages tab), and change the Sensor Key field to one of sound, co2, temperature, voc, battery, light, uv, humidity, pressure.
Once you have setup the system and followed all of the instructions, you should get MQTT messages on the platform several times a minute with data collected from the Thunderboard.
AddThunderboardDevice - Checks if a thunderboard device exists in the device table, if yes then sends authentication message back to gateway.
DetectAnomaly - Detects an anomaly in the last X MQTT Messages
ExampleDetectAnomaly - Runs anomaly detection against an example dataset, returns an array of Anomalies with metadata.
GetRecentAnomalies - This service retrieves the 10 most recent data points from the Anomalies collection
SaveAnomalyConfiguration - Ingests updated anomaly configuration, and saves to AnomalyConfiguration Collection
SendAlert - Fetches the anomaly row, checks for alerting configuration, and sends email, text alerts
AnomalyDetection - Core library for detecting anomalies in datasets
AnomalyDetectionConstants - Optional credentials for Alerting (SMS via Twilio, Email via Sendgrid)
jStat - Dependency on jstat-statistics-toolkit
SendGridEmail - Dependency on send-grid-email
TwilioSMS - Dependency on twilio-sms-library
NewThunderboardDevice - triggers on MQQT message published to +/status/+
SendAlertsTrigger - triggers when item created in Anomalies collection
AnomalyDetection - Portal for viewing and configuring live datafeeds, rules, and alerts
- SEND_GRID_API_URI
Creator: Robert Reinold Updated: 2017-10-02T00:00:00Z Version: v2.0 Tags: email, sendgrid, marketing, mail, api, rest, http
Usage:
- Create a free SendGrid Account.
- Log into your SendGrid account, and view the Settings > API Keys tab. Create an API Key with full access to "Mail Send" rights.
- Replace <SEND_GRID_API_KEY> with SendGrid API key
- Replace <ORIGIN_EMAIL_ADDRESS> with your desired email address. This will be the 'sender' of the email.
- Add 'SendGridEmail' as a dependency to your code services (Settings > Requires > Add)
- template
- AnomalyDetection()
Detects Anomalies with Moving Median Decomposition See attached whitepaper for Anomaly Detection for Predictive Maintenance https://files.knime.com/sites/default/files/inline-images/Anomaly_Detection_Time_Series_final.pdf
Run a self-training anomaly detection algorithm against a given dataset
- Twilio(user, pass, sourceNumber)
Sends a text message using Twilio's REST API.
- AddThunderboardDevice(body)
When the gateway detects a thunderboard via BLE, it checks to see if it has configuration values that match a device in the device table. If not it creates a device. This code service triggers when a MQQT message is sent for the above reason, and if there does exist a matching device then it sends a message back to the gateway authenticating the device so it can start gathering data.
- DetectAnomaly() ⇒
AnomalyVisual Detects an anomaly in the last X MQTT Messages Uses 'AnomalyConfiguration' config to check for a key in the JSON of each message. ex. 'sensor_key' will be set to 'temperature', so we know to pull the temperature key/value pair from a mqtt message like this: {"temperature":40,"humidity":31}
- ExampleDetectAnomaly() ⇒
Array.<Anomaly> Example logic for detecting anomalies in a dataset of sensor values
- SaveAnomalyConfiguration(anomalyConfiguration)
Ingests updated anomaly configuration, and saves to AnomalyConfiguration Collection
- SendAlert(item_id) ⇒
Object This is triggered by an Anomaly being recorded, and inserted into the Anomalies Collection
- Fetches the anomaly row
- Checks for alerting configuration
- Sends email, text alert
- Calibration
- Anomaly
- AnomalyVisual :
Object - self-explanatory ⇒
data This service retrieves the 10 most recent data points from the Anomalies collection.
Creator: Robert Reinold Updated: 2017-10-02T00:00:00Z Version: v2.0 Tags: email, sendgrid, marketing, mail, api, rest, http
Usage:
- Create a free SendGrid Account.
- Log into your SendGrid account, and view the Settings > API Keys tab. Create an API Key with full access to "Mail Send" rights.
- Replace <SEND_GRID_API_KEY> with SendGrid API key
- Replace <ORIGIN_EMAIL_ADDRESS> with your desired email address. This will be the 'sender' of the email.
- Add 'SendGridEmail' as a dependency to your code services (Settings > Requires > Add)
Detects Anomalies with Moving Median Decomposition See attached whitepaper for Anomaly Detection for Predictive Maintenance https://files.knime.com/sites/default/files/inline-images/Anomaly_Detection_Time_Series_final.pdf
Run a self-training anomaly detection algorithm against a given dataset
Kind: global function
Parameter: number[] dataset array of numbers
Use precomputed anomaly detection calibration profile This can be used to speed up performance for real-time anomaly detection
Kind: inner method of AnomalyDetection
AnomalyDetection~detect() ⇒ Array.<Anomaly>
Run the algorithm against the provided dataset.
Note: This is the most computation-heavy method in this library
Kind: inner method of AnomalyDetection
Returns: Array.<Anomaly> - anomalies - the list of anomalies found in dataset
Parameter: number strictnessOverride (optional) Set the strictness of the anomaly detection
AnomalyDetection~getCalibration() ⇒ Calibration
Fetch the computed anomaly detection calibration parameters
Kind: inner method of AnomalyDetection
Returns: Calibration - calibration
Sends a text message using Twilio's REST API.
Kind: global function
| Param | Type | Description |
|---|---|---|
| user | string |
Twilio API account ex. "BC218b72987d86855a5adb921370115a20" |
| pass | string |
Twilio API passcode ex. "4579ac6ba4fae7b452c03c64aeae40e7" |
| sourceNumber | string |
Origin phone number of text message, ex "(+1 512-713-2783)" |
Send SMS message
Kind: inner method of Twilio
| Param | Type | Description |
|---|---|---|
| text | string |
text body |
| recipientNumber | string |
Formatted phone number ex. "(+1 339-987-2816)" |
| callback | callback |
ex. function(err, data){} |
When the gateway detects a thunderboard via BLE, it checks to see if it has configuration values that match a device in the device table. This code service triggers when a MQQT message is sent for the above reason, and if there does exist a matching device then it sends a message back to the gateway authenticating the device so it can start gathering data.
Kind: global function
| Param | Type | Description |
|---|---|---|
| body | Object |
contains the MQQT status message sent from the gateway |
| body.status | string |
should always be "New" if it exists |
| body.deviceId | int |
is the name of the device specified in the adapter, should match the name of a device in the device table |
DetectAnomaly() ⇒ AnomalyVisual
Detects an anomaly in the last X MQTT Messages Uses 'AnomalyConfiguration' config to check for a key in the JSON of each message. ex. 'sensor_key' will be set to 'temperature', so we know to pull the temperature key/value pair from a mqtt message like this: {"temperature":40,"humidity":31}
Kind: global function
Returns: AnomalyVisual - data to visualize this anomaly detection
ExampleDetectAnomaly() ⇒ Array.<Anomaly>
Example logic for detecting anomalies in a dataset of sensor values
Kind: global function
Returns: Array.<Anomaly> - anomalies
Ingests updated anomaly configuration, and saves to AnomalyConfiguration Collection
Kind: global function
| Param | Type | Description |
|---|---|---|
| anomalyConfiguration | AnomalyConfiguration |
map of settings to update |
This is triggered by an Anomaly being recorded, and inserted into the Anomalies Collection
- Fetches the anomaly row
- Checks for alerting configuration
- Sends email, text alert
Kind: global function
Returns: Object - response - simple response of success or error
| Param | Type | Description |
|---|---|---|
| item_id | number |
the id of the anomaly in Anomalies collection to be retrieved |
Kind: global typedef
Properties
| Name | Type | Description |
|---|---|---|
| min | number |
min value within threshold |
| max | number |
max value within threshold |
| strictness | number |
how strict the threshold is against the dataset |
| medians | number |
intermediary dataset representing the moving medians |
| pointsPerWindow | number |
calibration, number of points per processing window. may be increased for larger datasets |
| numWindows | number |
calibration metric derived from pointsPerWindow |
Kind: global typedef
Properties
| Name | Type | Description |
|---|---|---|
| index | number |
index in the provided dataset |
| value | number |
value of the data point that is detected as an anomaly |
Kind: global typedef
Properties
| Name | Type | Description |
|---|---|---|
| toViz | Array.<VisualPoints> |
array of points to be displayed in graph in portal |
| anomalies | Array.<Anomaly> |
these are datapoints considered to be anomalies |
This service retrieves the 10 most recent data points from the Anomalies collection.
Kind: global typedef
Returns: data - response
Properties
| Name | Type | Description |
|---|---|---|
| CURRENTPAGE | number |
the results from the colleciton are all from one page |
| DATA | Array.<Object> |
each Object in the array contains: |
| above_threshold | boolean |
true if this datapoint is an anomaly |
| anomaly_timestamp | timestamp |
when the anomaly was detected |
| item_id | number |
uuid |
| sensor_key | string |
which sensor this anomaly is from |
| threshold_max | number |
if the value is higher than this then its an anomaly |
| threshold_min | number |
if the value is lower than this then its an anomaly |
| val | number |
the actual value retrieved from the sensor |