Skip to content
This repository has been archived by the owner on Nov 16, 2023. It is now read-only.

Commit

Permalink
updates
Browse files Browse the repository at this point in the history
  • Loading branch information
Scott Cate authored and Scott Cate committed Dec 11, 2018
1 parent a675561 commit 3f5ff7d
Show file tree
Hide file tree
Showing 257 changed files with 1,209 additions and 114,273 deletions.
50 changes: 45 additions & 5 deletions DAT - Getting the most out of your data/DAT10/README.md
@@ -1,11 +1,51 @@
# Dealing with a Massive Onset of Data Ingestion

Instructions and code to recreate the demo in the Microsoft Ignite | The Tour session titled [_Dealing with a Massive Onset of Data Ingestion_](./session).
How does a large company like Tailwind Traders build their data ingestion and processing pipelines, and chooses appropriate technologies to drive their business forward? This repo solves this problem by building a scalable and flexible end-to-end architecture that is capable of receiving massive amounts of data from variety of sources in real-time, and processing it.

# Edge Nodes
## What's Inside

A living repository of best practices and examples for developing [Azure IoT Edge](./edge).
This repo contains instructions and code to recreate the demo in the Microsoft Ignite | The Tour session titled _Dealing with a Massive Onset of Data Ingestion_.

# Data Generator
## Azure Services Used

An IoT Edge Module that [Generates Sample Data](./data-generator)
- Event Hubs
- IoT Hub
- Cosmos DB
- Azure Databricks
- Azure Blob Storage
- Azure Functions

## Prerequisites

* [Azure Subscription](https://azure.microsoft.com/free/?WT.mc_id=MSIgniteTheTour-github-dat10)
* [Docker](https://docs.docker.com/install/)
* [Azure Cli](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest&WT.mc_id=MSIgniteTheTour-github-dat10)
* [Azure Functions Core Tools](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local#v2?WT.mc_id=MSIgniteTheTour-github-dat10)
* [Python 2.7x OR Python 3.x](https://www.python.org/downloads/)

## Setup Instructions

All of these steps are required to be completed in order before proceeding with the dempo.

1. Create the cash register data generator with the following instructions found [here](DataGenerator/)
2. Create the Azure function with the following instructions found [here](ThresholdEndpoint/)
3. Create the data ingestion layer with the following instructions found [here](Data-ingestion-and-processing/)


## Demo Instructions

Upon completing the Setup Instructions above, you can log into the Databricks instance that was created and follow the instructions embedded in the [deployed notebook](/Data-ingestion-and-processing/code/DataProcessing.dbc).

## Learn More/Resources

[IoT Hub](https://docs.microsoft.com/en-us/azure/iot-hub/about-iot-hub?WT.mc_id=MSIgniteTheTour-github-dat10)

[Microsoft Learn: Enable reliable messaging for Big Data applications using Azure Event Hubs](https://docs.microsoft.com/en-us/learn/modules/enable-reliable-messaging-for-big-data-apps-using-event-hubs/index?WT.mc_id=MSIgniteTheTour-github-dat10)

[Azure Databricks](https://docs.microsoft.com/en-us/azure/azure-databricks/?WT.mc_id=MSIgniteTheTour-github-dat10)

[Azure-IoT-Edge-Bogus-Data-Generator](http://aka.ms/iotdatagenerator)

[Azure Functions](https://docs.microsoft.com/en-us/azure/azure-functions/?WT.mc_id=MSIgniteTheTour-github-dat10)

[Cosmos DB](https://docs.microsoft.com/en-us/azure/cosmos-db/?WT.mc_id=MSIgniteTheTour-github-dat10)

This file was deleted.

This file was deleted.

This file was deleted.

Binary file not shown.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

0 comments on commit 3f5ff7d

Please sign in to comment.