This repository has been archived by the owner on Nov 16, 2023. It is now read-only.
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Scott Cate
authored and
Scott Cate
committed
Dec 11, 2018
1 parent
a675561
commit 3f5ff7d
Showing
257 changed files
with
1,209 additions
and
114,273 deletions.
There are no files selected for viewing
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,51 @@ | ||
# Dealing with a Massive Onset of Data Ingestion | ||
|
||
Instructions and code to recreate the demo in the Microsoft Ignite | The Tour session titled [_Dealing with a Massive Onset of Data Ingestion_](./session). | ||
How does a large company like Tailwind Traders build their data ingestion and processing pipelines, and chooses appropriate technologies to drive their business forward? This repo solves this problem by building a scalable and flexible end-to-end architecture that is capable of receiving massive amounts of data from variety of sources in real-time, and processing it. | ||
|
||
# Edge Nodes | ||
## What's Inside | ||
|
||
A living repository of best practices and examples for developing [Azure IoT Edge](./edge). | ||
This repo contains instructions and code to recreate the demo in the Microsoft Ignite | The Tour session titled _Dealing with a Massive Onset of Data Ingestion_. | ||
|
||
# Data Generator | ||
## Azure Services Used | ||
|
||
An IoT Edge Module that [Generates Sample Data](./data-generator) | ||
- Event Hubs | ||
- IoT Hub | ||
- Cosmos DB | ||
- Azure Databricks | ||
- Azure Blob Storage | ||
- Azure Functions | ||
|
||
## Prerequisites | ||
|
||
* [Azure Subscription](https://azure.microsoft.com/free/?WT.mc_id=MSIgniteTheTour-github-dat10) | ||
* [Docker](https://docs.docker.com/install/) | ||
* [Azure Cli](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest&WT.mc_id=MSIgniteTheTour-github-dat10) | ||
* [Azure Functions Core Tools](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local#v2?WT.mc_id=MSIgniteTheTour-github-dat10) | ||
* [Python 2.7x OR Python 3.x](https://www.python.org/downloads/) | ||
|
||
## Setup Instructions | ||
|
||
All of these steps are required to be completed in order before proceeding with the dempo. | ||
|
||
1. Create the cash register data generator with the following instructions found [here](DataGenerator/) | ||
2. Create the Azure function with the following instructions found [here](ThresholdEndpoint/) | ||
3. Create the data ingestion layer with the following instructions found [here](Data-ingestion-and-processing/) | ||
|
||
|
||
## Demo Instructions | ||
|
||
Upon completing the Setup Instructions above, you can log into the Databricks instance that was created and follow the instructions embedded in the [deployed notebook](/Data-ingestion-and-processing/code/DataProcessing.dbc). | ||
|
||
## Learn More/Resources | ||
|
||
[IoT Hub](https://docs.microsoft.com/en-us/azure/iot-hub/about-iot-hub?WT.mc_id=MSIgniteTheTour-github-dat10) | ||
|
||
[Microsoft Learn: Enable reliable messaging for Big Data applications using Azure Event Hubs](https://docs.microsoft.com/en-us/learn/modules/enable-reliable-messaging-for-big-data-apps-using-event-hubs/index?WT.mc_id=MSIgniteTheTour-github-dat10) | ||
|
||
[Azure Databricks](https://docs.microsoft.com/en-us/azure/azure-databricks/?WT.mc_id=MSIgniteTheTour-github-dat10) | ||
|
||
[Azure-IoT-Edge-Bogus-Data-Generator](http://aka.ms/iotdatagenerator) | ||
|
||
[Azure Functions](https://docs.microsoft.com/en-us/azure/azure-functions/?WT.mc_id=MSIgniteTheTour-github-dat10) | ||
|
||
[Cosmos DB](https://docs.microsoft.com/en-us/azure/cosmos-db/?WT.mc_id=MSIgniteTheTour-github-dat10) |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 0 additions & 2 deletions
2
DAT - Getting the most out of your data/DAT10/data-generator/.gitignore
This file was deleted.
Oops, something went wrong.
43 changes: 0 additions & 43 deletions
43
DAT - Getting the most out of your data/DAT10/data-generator/.vscode/launch.json
This file was deleted.
Oops, something went wrong.
16 changes: 0 additions & 16 deletions
16
DAT - Getting the most out of your data/DAT10/data-generator/README.MD
This file was deleted.
Oops, something went wrong.
Binary file removed
BIN
-242 KB
DAT - Getting the most out of your data/DAT10/data-generator/assets/demo.png
Binary file not shown.
58 changes: 0 additions & 58 deletions
58
DAT - Getting the most out of your data/DAT10/data-generator/deployment.template.json
This file was deleted.
Oops, something went wrong.
34 changes: 0 additions & 34 deletions
34
... of your data/DAT10/data-generator/modules/azure-iot-edge-bogus-data-generator/.gitignore
This file was deleted.
Oops, something went wrong.
97 changes: 0 additions & 97 deletions
97
...data/DAT10/data-generator/modules/azure-iot-edge-bogus-data-generator/BogusDataFactory.cs
This file was deleted.
Oops, something went wrong.
51 changes: 0 additions & 51 deletions
51
...DAT10/data-generator/modules/azure-iot-edge-bogus-data-generator/DesiredPropertiesData.cs
This file was deleted.
Oops, something went wrong.
17 changes: 0 additions & 17 deletions
17
... of your data/DAT10/data-generator/modules/azure-iot-edge-bogus-data-generator/Dockerfile
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.