Contoso Auto is a high value cargo logistics organization that is collecting vehicle and package telemetry data and wants to use Azure Cosmos DB to rapidly ingest and store this data in its raw form, do some processing in near real-time to generate insights to support several business objectives and surface these to the most appropriate user communities within the organization. It is a fast growing organization and wants to be able to scale and manage the associated cost of its chosen technology to enable it to cope with its explosive growth and the inherent seasonality of the logistics business. This scenario includes applicability to both the vehicle telemetry and logistics use cases by focusing on trucking and inclusion of cargo sensing data. This additionally allows for many representative customer analytics scenarios.
Contoso would like to leverage Azure Cosmos DB as the core repository for its hot data path and leverage the Azure Cosmos DB Change Feed as a means to drive a solid and robust event sourcing architecture that would allowing Contoso developers to quickly enhance the solution. This achieved using a robust and agile serverless approach by leveraging events published by the Change Feed that reflect the state changes within the application (database).
Ultimately Contoso would surface the raw and derived insights data to its users in one of three roles:
-
Logistics Operations personnel who are interested in the current state of the vehicles and cargo logistics and who would use a web app to quickly understand the status of any single vehicle or piece of cargo, be notified of alerts as well as load vehicle and cargo meta data into the system. What they would like to see on the dashboard are various visualizations of detected anomalies, like engines overheating, abnormal oil pressure, and aggressive driving.
-
Management and Customer Reporting personnel who would like to be in a position to see the current state of the vehicle fleet and customer consignment level information presented in on a Power BI report that automatically updates with new data as it flows in after being processed. What they would like to see are reports on bad driving behavior by driver and using visual components such as a map to show anomalies related to cities or areas, as well as various charts and graphs depicting aggregate fleet and consignment information in a clear way. ` In this experience, you will use Azure Cosmos DB to ingest streaming vehicle telemetry data as the entry point to a near real-time analytics pipeline built on Cosmos DB with Analytical Storage as well as Azure Synapse, Azure Functions, Azure Web Apps, Logic Apps, Azure IoT Hub, Azure Event Hub, Azure Stream Analytics, and Power BI.
Below is a diagram of the solution architecture you will build in this lab. Please study this carefully, so you understand the whole of the solution as you are working on the various components.
-
Data ingest, event processing, and storage:
The solution for the IoT scenario centers around Cosmos DB, which acts as the globally-available, highly scalable data storage for streaming event data, fleet, consignment, package, and trip metadata, and aggregate data for reporting. Vehicle telemetry data flows in from the data generator, through registered IoT devices in IoT Hub, where an Azure function processes the event data and inserts it into a telemetry container in Cosmos DB.
-
Trip processing with Azure Functions:
The Cosmos DB change feed triggers two separate Azure functions. The functions each manage their own checkpoint so they can process the same incoming data without conflicting with each other. One function processes the vehicle telemetry, aggregating the batch data and updating the trip and consignment status in the metadata container based on odometer readings and whether the trip is running on schedule. This function also triggers a Logic App to send email alerts when trip milestones are reached. The other function sends the event data to Event Hubs, which in turn triggers Stream Analytics to execute time window aggregate queries.
-
Stream processing, dashboards, and reports:
The Stream Analytics queries output vehicle-specific aggregates to the Cosmos DB metadata container, and overall vehicle aggregates to Power BI to populate its real-time dashboard of vehicle status information. A Power BI Desktop report is used to display detailed vehicle, trip, and consignment information, pulled directly from the Cosmos DB metadata container. It also displays batch battery failure predictions, pulled from the maintenance container.
-
Advanced analytics and ML model training:
Azure Synapse connects to Azure Cosmos DB Analytical Store via Synapse Link for Cosmos DB to retrieve daily trip information, transform it, write it to a data set that Azure Machine Learning can use to train batch and real-time prediction models, and then write batch predictions back to Cosmos DB. A Spark cluster and Python notebooks are used for data movement and transformation between Azure Cosmos DB Analytical Store and Azure Data Lake Store.
Azure Machine Learning is used to train a machine learning model to predict vehicle battery failure based on historic information. Batch and real-time inferencing pipelines are published. The real-time inference pipeline is published to an Azure Kubernetes Service (AKS) cluster, where it is called by the web app when a user requests a battery maintenance prediction.
-
Fleet management web app, security, and monitoring:
A Web App allows Contoso Auto to manage vehicles and view consignment, package, and trip information that is stored in Cosmos DB. The Web App is also used to make real-time battery failure predictions while viewing vehicle information. Azure Key Vault is used to securely store centralized application secrets, such as connection strings and access keys, and is used by the Function Apps and the Web App. Finally, Application Insights provides real-time monitoring, metrics, and logging information for the Function Apps and Web App.
There are two separate deployments of this scenario: a hands-on lab, and a demo deployment. Each deployment results in the same final set of deployed Azure resources.
The hands-on lab starts with a minimal set of automated Azure resource deployments. Learners will then step through several hands-on deployment and configuration exercises to complete the overall lab.
The demo deployment automates all Azure resource deployment steps, with a minimum of manual steps required to complete the overall deployment. This deployment results in the complete deployment as quickly as possible to support interactive exploration and demonstrations.
To go through either the hands-on lab or the demo deployment, please follow all the steps in order.