Setting up real time data pipelines in Azure, with Azure Stream Analytics allows one to quickly scale and perform advanced analytics on a moving data stream. The following repository demonstrates how we easily can utilize various Azure services to easily spin up a data pipeline to stream data from multiple sources using serverless functionality in Azure.
- Azure Logic Apps
- Azure Service Bus
- Azure EventHub
- Azure Stream Analytics
- Azure DevOps
- Power BI
- Raspberry Pi
- GrovePi
There are endless potential use-cases in which this tech-stack can be proven useful. To mention a few, we could potentially stream:
- Social media and RSS feeds to determine sentiment and overall success of a newly released feature or product
- Data from live traffic cameras to determine current conditions and measure improvements after roadwork
- Data from live traffic cameras to determine potential benefits/losses of introducing or removing toll roads
- Financial transactions to determine fraudulent activities
- IoT sensors of various kinds in the field to determine and predict air quality in different cities across the world
Please refer to the following blog posts, presentations and recorded talks for further information:
- Getting started with Azure Stream Analytics
- Near Real-Time Data Streaming with Azure Stream Analytics
- Real-Time Data Streaming with Azure Stream Analytics
- PowerPoint presentation
This repo demonstrates the ability to stream Twitter data based on keywords or specific users utilizing server-less functionality such as Azure Logic Apps, EventHub and Stream Analytics. The egressed and analyzed data can be visualized in a Power BI dashboard.
Please follow the following guide to set up the infrastructure with Azure Resource Management Templates and run semantic analysis on Twitter Feeds
This repo demonstrates the ability to stream IoT sensor data from a Raspberry PI to the cloud, utilizing an EventHub, a Stream Analytics job, an Azure Service Bus and an Azure Logic App. The sensor data can be viewed in a Power BI dashboard, but there is also built in functionality to demonstrate how easy it is to set up your own burglar alarm. Any motion detected by the Raspberry PI's sensors will put a message in a service bus queue that will be picked up by an Azure Logic App which in turn will send a notification e-mail to a given e-mail address. The sample demonstrates the LAG functionality in particular, but also how to use reference data to enrich the stream.
Please follow the following guide to set up the infrastructure with Azure Resource Management Templates and start streaming IoT data from a Raspberry Pi
See anything you want to improve? Do you want to build out the existing code base? Don't heistate to open a PR! Please read our contributing guidelines