This repository contains complete and easy to use examples that demonstrate the power of Apache Storm on Microsoft Azure HDInsight.
The .Net examples are applicable to all HDInsight Windows Storm clusters.
The Java examples are applicable to :
- 3.3 and below versions of HDInsight Windows
- 3.4 and below versions of HDInsight Linux Storm clusters.
For version 3.5+ Linux based HDInsight Storm clusters please look under HDI3.5 folder.
Official page - Apache Storm for Microsoft Azure HDInsight
Getting Started page - Getting Started with HDInsight Storm
- End-to-end examples - Read more about them below
- EventCountExample - Scalability benchmark for Microsoft Azure EventHubs/Apache Kafka on Microsoft Azure HDInsight
- IotExample - Internet of things - connected cars scenario
- RealTimeETLExample - Web request aggregation into HBase
- SCPNetExamples - Examples that show API usage of SCP.Net
- lib - Java dependencies
- scripts - Automation scripts
- Templates - Azure services templates
- tools - Helper tools
Running the examples
Each of the examples folder contains a
run.bat that does the following:
- Prepare (using
- Creation of required services (and resources) in Azure
- Updating project properties and templates
- Build (using
- Building the source code and use the properties created in previous step
- Execute (using
- Upload jars and create topology packages (as needed)
- Submitting the topologies to the HDInsight Storm cluster
An existing run configuration is first detected to provide idem-potency in creation of Azure resources. This allows you to re-run the
run.bat in case of some unexpected failure.
Building the examples
If you just want to build the source code you can use the
build.bat in each example folder to build that example.
Re-running the example
If you modified some source and wish to re-deploy the topologies, you can build using
build.bat and submit the topologies using
There are multiple end-to-end examples included in this repository that showcase different topologies that you can build and run against HDInsight Storm. HDInsight provides users the option to write applications in language of their choice and exposes Storm REST APIs as well as additional web interfaces for remote topology submission and management.
Internet of Things Example (Java)
An internet of things example that demonstrates connected cars scenarios. The example showcases the reading of vehicle events from Event Hub, referencing Azure DocumentDB for vehicular data and then finally storing the computations in Azure Storage using the WASB driver on HDInsight clusters.
Read more about this example here: IoT example using Azure EventHubs, Storm, Azure DocumentDB, Azure Storage/WASB
EventHubs/Kafka Scalability Example (Java & SCP.Net)
A scalability benchmark that demonstrates throughput from EventHub/Kafka and stores the event counts in SQL Azure.
Read more about this example here: Event Count example using Azure EventHubs, Storm, SQL Azure
Real Time ETL Example (SCP.Net)
A real time ETL example with EventHubs, Storm and HBase that has two topologies.
- The first topology Event Sender Topology generates Web Request Logs and pushes them to EventHubs.
- The second topology Event Aggregation Topology reads from EventHubs, calculates aggregations("count() group by") and then stores the computations in a HDInsight HBase cluster.
Read more about this example here: Real Time ETL example using Azure EventHubs, Storm, HBase & SCP.Net
Check out more SCP.Net Examples
This directory now contains many SCP.Net examples that show the APIs available in SCP.Net NuGet package. Follow the instructions in the above directory to deploy these examples to your HDInsight Storm cluster.
HybridTopologyHostMode in SCPNetExamples/HybridTopologyHostMode has examples on all the different types of hybrid modes.
Topology, Spout Bolt templates
Templates to connect to various Azure services:
- SQL Azure
- Azure Websites (SignalR)
Azure Services and HDInsisght Helper PowerShell Scripts
- Looking for some Azure PowerShell scripts that you can use individually or in an automation?
- Azure PowerShell scripts to create HDInsight clusters
- [HDInsight] (http://azure.microsoft.com/en-us/documentation/services/hdinsight/)
- [Azure Event Hubs] (http://azure.microsoft.com/en-us/services/event-hubs/)
- [DocumentDB] (http://azure.microsoft.com/en-us/services/documentdb/)
- [SQL Azure] (http://azure.microsoft.com/en-us/services/sql-database/)