This project is part of the 'IBM Hybrid Analytics and Big Data Architecture' reference architecture implementation, available at https://github.com/ibm-cloud-architecture/refarch-analytics. This set of projects presents an end to end solution to enable predictive maintenance capabilities on manufacturing assets. The problem space is related to continuous operation and service on manufacturing asset like Electrical Submersible Pump, but any assets with sensors can be considered with the same of solution.
The use case is also adaptable and the architecture, solution components can be used for a security threat analysis on a park of devices or assets connected to a company intranet: real time events come from device that need to be aggregated and correlated and analytics run can be performed on historical data to assess security risk. Alert can be published to dashboard user interface.
We are presenting best practices around data management, real-time streaming, cassandra high availability, microservices and serverless implementation.
- Use case
- System Context to present the solution components
- Deployment to kubernetes cluster like IBM Cloud Private
- Related sub projects - repositories
- Event consumers
- Event producers simulator to simulate pump events for demonstration purpose.
- Angular 6 user interface to present dashboard to present a mix of static and real time data.
- Dashboard BFF to present a mix of static and real time data.
- Asset management microservice used to expose REST api in front of Cassandra persistence
- Demonstration script
- Analytics model
- Future readings
The adoption of IoT, smart devices, in manufacturing industry brings opportunity to predict future maintenance on high-cost equipment before failure. The cost control for maintenance operation combined with optimizing device utilization are continuous challenges engineers are facing. For IT architects, the challenge is to address how to prepare to support adopting artificial intelligence capacity to support new business opportunities? How to support real-time data analytics at scale combined with big data and microservice architecture?
A set of geographically distributed electrical submersible pumps (can apply to any manufacturing IoT equipment) are sending a data stream about important physical measurements that need to be processed in real-time and persisted in big data storage. By adding traditional analytics combined with unstructured data as field engineer's reports, it is possible to build a solution that delivers a risk of failure ratio, in real time, from measurements received.
The solution combines key performance indicators aggregation, real-time reporting to a web-based dashboard user interface component, risk scoring microservice, and big data sink used by data scientists to develop and tune analytics and machine learning models.
Data are continuously persisted in a document oriented database, we selected Cassandra as a data sink. The real time event processing is supported by Kafka and Kafka streaming. The microservices are done in Java, one in microprofile and one in Java. The data science work is done using ICP for Data and data science experience.
The solution gives visibility to analysts and executive about the real time status of the devices in the grids, with an aggregate view of the ones at risk. The device operation was increased by 15% and the unpredictable failure rate decreased by 85%. The model added diagnostic capabilities to help field engineers to deliver better maintenance.
The processing starts by the continuous event flow emitted by a set of monitored devices. The event platform offers pub-subs capabilities and flow processing to aggregate and correlate events so dash board monitoring can be implement on the stateful operators. Data sink are used to keep Asset information, physical measurements over time and maintenance reports. Risk scoring service is deployed as a REST operation, build from a model developed by analytics and machine learning capabilities. Assets are exposed via microservice.
-
The application logic is split between the backend for frontend components, and the different microservices or streaming operators. The BFF is a web app exposing a user interface and the business logic to serve end users. For example when a new device or pump is added to the grid, a record is pushed to the user interface. All the real time metrics per device are also visible on the user interface. The supporting project is the Dashboard BFF and ...
-
The user interface is done using Angular 6 and aim to present a dashboard of the pump allocation and real time metrics. The wireframe looks like:
and the project is under the asset-dashboard-ui folder.
-
Manage CRUD operation on the assets. See the Asset manager microservice code.
-
Pump Simulator is a java program running on developer laptop or on a server but external to ICP. The approach is to address communication to brokers deployed on ICP. The code and guidance are in the asset-event-producer project.
The following diagram illustrates the IBM Cloud Private deployment, we are using in this solution. You will find the same components as in the system context above, with added elements for data management and data scientists using ICP for Data.
The pump simulator is a standalone Java application you can run on your computer to simulate n pumps. See the producer project.
We propose two possible deployments: one for quick validation in a development laptop (tested on Mac) and one on IBM Cloud Private cluster (tested on 2.1.0.3 and 3.1).
-
Clone this project to get all the kubernetes deployment files and source code of the different components.
-
Clone the Event Driven Architecture reference project: EDA to get Zookeeper and Kafka deployment manifests. As an alternate solution you can use the IBM Event Streams Helm chart from the ICP Catalog.
git clone https://github.com/ibm-cloud-architecture/refarch-eda.git-
Clone the asset management microservice using the microprofile branch implementation.
git clone https://github.com/ibm-cloud-architecture/refarch-asset-manager-microservice.git git checkout microprofile
-
Access to a kubernetes deployment for development, for example on Mac, we use Docker Edge distribution. You can use this article to install Docker Edge and enable kubernetes.
For test and 'production' deployments you need to have access to a kubernetes cluster like IBM Cloud Private.
- Login to your cluster We are providing multiple scripts to support connection and configuration validation:
- To connect to your cluster:
./scripts/connectToCluster.sh - To validate your dependencies:
./scripts/validateConfig.sh. It may create the 'greencompute' namespace if it does not exist.
For your local environment install Zookeeper and Kafka using our development manifests:
Read instructions in this article
Read instructions in this article
For production we recommend to install IBM Events Streams
Read instructions in this article
There is no Cassandra helm chart currently delivered with ICP Helm catalog. The asset manager microservice has a helm chart to install Cassandra on ICP, and an helm chart for the microservice itself. In fact there is an umbrella chart to deploy both in one install. The instructions can be read here. But it basically do the following helm install under the folder of refarch-asset-manager-microservice
helm install --name assetmanager asset/assetmanager --namespace greencompute --tls
We are also summarizing Cassandra concepts and some installation considerations in this article. In the readme, we also describe the potential architecture decisions around deploying Cassandra for high availability.
When the pods are up and running use the following commands to create the needed keyspace and tables for the solution to run.
The following steps are manual. We need to deploy the asset manager microservice first, then the different event consumers.
The docker image is pushed to docker hub, therefore the manifests under refarch-asset-manager-microservice/manifests is using this image.
See instructions in the project repository to deploy it using helm.
To deploy the BFF: the scripts and manifests are under the asset-dashboard-bff folder of this project. We document the deployment and build process in these instructions.
The project https://github.com/ibm-cloud-architecture/refarch-asset-manager-microservice has one script to do a curl post with json file representing pumps. You can change the URL to match your asset manager microservice endpoint, and then deploy the pump once the asset manager is deployed.
$ cd scripts
$ ./addAsset.sh pumpDAL01.json
$ ./addAsset.sh pumpHOU1.json
$ ./addAsset.sh pumpLA1.json
$ ./addAsset.sh pumpLA2.json
$ ./addAsset.sh pumpLA3.json
$ ./addAsset.sh pumpND1.json
$ ./getAssets.shThe dashboard reports the imported pumps:
We have developed two types of asset injector. We propose to use a function as service as the base one. So see the instruction to install kubeless and the consume asset function in this note
The second asset injector is part of the asset-consumer project and is the Springboot app which can be built and executed with the commands:
mvn compile
mvn exec:java@AssetInjectorBe sure to modify the config/config.properties file to match the server endpoint for Kafka and Asset Manager Microservice.
Finally the pump simulator is a standalone java application used to produce different types of event. It does not need to be deployed to kubernetes. It can run in two modes:
- add new asset: this is when a pump is added to a location.
mvn compile
mvn exec:java@Simulator- run pump metrics
mvn compile
mvn exec:java@Simulator -Dexec.args="asset-topic gc-kafka-0.gc-kafka-hl-svc.greencompute.svc.cluster.local event PDrop 2000 4 pump01"The diagram below presents the deployment of the solution components as well as the Zookeeper, Kafka and Cassandra products deployed inside kubernetes:
- For high availability we need three masters, three proxies, 3 managers and at least 6 workers.
- Cassandra is deployed with 3 replicas and uses NFS based persistence volume so leverage shareable filesystems.
- Kafka is deployed with 3 replicas with anti affinity to avoid to have two pods on same node and also on the same node as Zookeeper's ones.
- Zookeeper is deployed with 3 replicas with anti affinity to avoid to have two pods on same node and on the same node as Kafka. This constraint explains the 6 workers.
- The following components of the solution are deployed with at least 3 replicas: Asset manager microservice, dashboard BFF.
As we are deploying different solutions into kubernetes we group Troubleshooting notes here and a technology summary here
- Lead development Jerome Boyer
- Amaresh Rajasekharan for the data science part
- Hemankita Perabathini for the asset management microservice and ICP
- Zach Silverstein
Please contact me for any questions.