Skip to content

Commit

Permalink
Merge pull request #6385 from samlamberson/brbq
Browse files Browse the repository at this point in the history
User Story 1915435 - Solution Ideas - Batch # 43
  • Loading branch information
v-ccolin committed Jun 14, 2022
2 parents 3ed1f37 + fcb39f9 commit cbfc21e
Show file tree
Hide file tree
Showing 10 changed files with 50 additions and 30 deletions.
Original file line number Diff line number Diff line change
@@ -1,18 +1,20 @@
<!-- cSpell:ignore KEDA deadletter -->
[!INCLUDE [header_file](../../../includes/sol-idea-header.md)]

This solution idea shows a variation of a serverless event-driven architecture using Azure Event Hubs and Azure Functions to ingest and process a stream of data. Once de-batched and filtered, the results are written to a database for storage and future review.
This solution idea shows a variation of a serverless event-driven architecture using Azure Event Hubs and Azure Functions to ingest and process a stream of data. The results are written to a database for storage and future review once de-batched and filtered.

To learn more about the basic concepts, considerations, and approaches for serverless event processing, consult the [Serverless event processing](../../reference-architectures/serverless/event-processing.yml) reference architecture.

## Potential use cases

A popular use case for implementing an end-to-end event stream processing pattern includes the Event Hubs streaming ingestion service to receive and process events per second using a de-batching and transformation logic implemented with highly scalable, event hub&#8211;triggered functions.
A popular use case for implementing an end-to-end event stream processing pattern includes the Event Hubs streaming ingestion service to receive and process events per second using a de-batching and transformation logic implemented with highly scalable, event hub-triggered functions.

## Architecture

![Diagram showing the data flow and key processing points in the architecture described in this article](../media/serverless-event-processing-filtering-diagram.png)

### Dataflow

1. Events arrive at the Input Event Hub.
1. The De-batching and Filtering Azure Function is triggered to handle the event. This step filters out unwanted events and de-batches the received events before submitting them to the Output Event Hub.
1. If the De-batching and Filtering Azure Function fails to store the event successfully, the event is submitted to the Deadletter Event Hub 1.
Expand All @@ -22,9 +24,17 @@ A popular use case for implementing an end-to-end event stream processing patter

### Components

- [Event Hubs](https://azure.microsoft.com/services/event-hubs/) ingests the data stream. Event Hubs is designed for high-throughput data streaming scenarios.
- [Azure Functions](https://azure.microsoft.com/services/functions/) is a serverless compute option. It uses an event-driven model, where a piece of code (a *function*) is invoked by a trigger.
- [Azure Cosmos DB](https://azure.microsoft.com/services/cosmos-db/) is a multi-model database service that is available in a serverless, consumption-based mode. For this scenario, the event-processing function stores JSON records, using the [Cosmos DB SQL API](/azure/cosmos-db/introduction).
- [Event Hubs](https://azure.microsoft.com/services/event-hubs) ingests the data stream. Event Hubs is designed for high-throughput data streaming scenarios.
- [Azure Functions](https://azure.microsoft.com/services/functions) is a serverless compute option. It uses an event-driven model, where a piece of code (a *function*) is invoked by a trigger.
- [Azure Cosmos DB](https://azure.microsoft.com/services/cosmos-db) is a multi-model database service that is available in a serverless, consumption-based mode. For this scenario, the event-processing function stores JSON records, using the [Cosmos DB SQL API](/azure/cosmos-db/introduction).

## Contributors

*This article is maintained by Microsoft. It was originally written by the following contributors.*

Principal author:

* [Rajasa Savant](https://www.linkedin.com/in/rajasa-savant-72645728) | Senior Software Development Engineer

## Next steps

Expand All @@ -35,8 +45,8 @@ A popular use case for implementing an end-to-end event stream processing patter

## Related resources

- [Azure Event Hubs documentation](/azure/event-hubs/)
- [Azure Event Hubs documentation](/azure/event-hubs)
- [Introduction to Azure Functions](/azure/azure-functions/functions-overview)
- [Azure Functions documentation](/azure/azure-functions/)
- [Azure Functions documentation](/azure/azure-functions)
- [Overview of Azure Cosmos DB](/azure/cosmos-db/introduction)
- [Choose an API in Azure Cosmos DB](/azure/cosmos-db/choose-api)
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ metadata:
description: A serverless event-driven architecture using Azure Event Hubs and Azure Functions to ingest and filter a stream of data for database storage.
author: rasavant-ms
ms.author: rasavant
ms.date: 06/25/2021
ms.date: 06/13/2022
ms.topic: conceptual
ms.service: architecture-center
ms.subservice: solution-idea
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,14 @@ To learn more about the basic concepts, considerations, and approaches for serve

## Potential use cases

A popular use case for implementing an end-to-end event stream processing pattern includes the Event Hubs streaming ingestion service to receive and process events per second using a de-batching and transformation logic implemented with highly scalable, event hub&#8211;triggered functions.
A popular use case for implementing an end-to-end event stream processing pattern includes the Event Hubs streaming ingestion service to receive and process events per second using a de-batching and transformation logic implemented with highly scalable, event hub-triggered functions.

## Architecture

[![Diagram showing data flowing into a VNet and then being processed by the architecture described in this article.](../media/serverless-event-processing-private-link-diagram.png)](../media/serverless-event-processing-private-link-diagram.png#lightbox)

### Dataflow

1. VNet integration is used to put all Azure resources behind [Azure Private Endpoints](/azure/private-link/private-endpoint-overview).
1. Events arrive at the Input Event Hub.
1. The De-batching and Filtering Azure Function is triggered to handle the event. This step filters out unwanted events and de-batches the received events before submitting them to the Output Event Hub.
Expand All @@ -27,9 +29,17 @@ A popular use case for implementing an end-to-end event stream processing patter
### Components

- [Azure Private Endpoint](/azure/private-link/private-endpoint-overview) is a network interface that connects you privately and securely to a service powered by Azure Private Link. Private Endpoint uses a private IP address from your VNet, effectively bringing the service into your VNet.
- [Event Hubs](https://azure.microsoft.com/services/event-hubs/) ingests the data stream. Event Hubs is designed for high-throughput data streaming scenarios.
- [Azure Functions](https://azure.microsoft.com/services/functions/) is a serverless compute option. It uses an event-driven model, where a piece of code (a *function*) is invoked by a trigger.
- [Azure Cosmos DB](https://azure.microsoft.com/services/cosmos-db/) is a multi-model database service that is available in a serverless, consumption-based mode. For this scenario, the event-processing function stores JSON records, using the [Cosmos DB SQL API](/azure/cosmos-db/introduction).
- [Event Hubs](https://azure.microsoft.com/services/event-hubs) ingests the data stream. Event Hubs is designed for high-throughput data streaming scenarios.
- [Azure Functions](https://azure.microsoft.com/services/functions) is a serverless compute option. It uses an event-driven model, where a piece of code (a *function*) is invoked by a trigger.
- [Azure Cosmos DB](https://azure.microsoft.com/services/cosmos-db) is a multi-model database service that is available in a serverless, consumption-based mode. For this scenario, the event-processing function stores JSON records, using the [Cosmos DB SQL API](/azure/cosmos-db/introduction).

## Contributors

*This article is maintained by Microsoft. It was originally written by the following contributors.*

Principal author:

* [Rajasa Savant](https://www.linkedin.com/in/rajasa-savant-72645728) | Senior Software Development Engineer

## Next steps

Expand All @@ -46,8 +56,8 @@ A popular use case for implementing an end-to-end event stream processing patter
- [Create an Azure Private Endpoint using Azure PowerShell](/azure/private-link/create-private-endpoint-powershell)
- [Create a Private Endpoint using Azure CLI](/azure/private-link/create-private-endpoint-cli)
- [Create a Private Endpoint by using an ARM template](/azure/private-link/create-private-endpoint-template)
- [Azure Event Hubs documentation](/azure/event-hubs/)
- [Azure Event Hubs documentation](/azure/event-hubs)
- [Introduction to Azure Functions](/azure/azure-functions/functions-overview)
- [Azure Functions documentation](/azure/azure-functions/)
- [Azure Functions documentation](/azure/azure-functions)
- [Overview of Azure Cosmos DB](/azure/cosmos-db/introduction)
- [Choose an API in Azure Cosmos DB](/azure/cosmos-db/choose-api)
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ metadata:
description: A serverless event-driven architecture in a VNet with private endpoints that ingests and processes a stream of data, then writes the results to a database.
author: rasavant-ms
ms.author: rasavant
ms.date: 06/25/2021
ms.date: 06/13/2022
ms.topic: conceptual
ms.service: architecture-center
ms.subservice: solution-idea
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ This solution address the capability to deliver agile development architecture t

## Architecture

![Architecture diagram](../media/sharepoint-farm-devtest.png)
![Architecture diagram shows resource groups of storage, network, and virtual machines to SharePoint, Load Balancer, V P N gateway and finally to users.](../media/sharepoint-farm-devtest.png)
*Download an [SVG](../media/sharepoint-farm-devtest.svg) of this architecture.*

### Data flow
### Dataflow

1. Create resource group for the storage, network, and virtual machine, plus other dependent elements.
1. Create a virtual network to host the virtual machines and load balancers for the deployment. Ensure the network has appropriate Network Security Groups implement to protect network traffic flow.
Expand Down
6 changes: 3 additions & 3 deletions docs/solution-ideas/articles/sharepoint-farm-devtest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ metadata:
titleSuffix: Azure Solution Ideas
description: Deploy a SharePoint farm for development testing. Use Azure Active Directory, SQL Server, and SharePoint resources for this agile development architecture.
author: EdPrice-MSFT
ms.author: pnp
ms.date: 12/16/2019
ms.author: edprice
ms.date: 06/14/2022
ms.topic: conceptual
ms.service: architecture-center
ms.subservice: solution-idea
Expand All @@ -21,7 +21,7 @@ metadata:
- azure sharepoint development environment
- sharepoint farm solution
- interactive-diagram
- "https://azure.microsoft.com/solutions/architecture/sharepoint-farm-devtest/"
- "https://azure.microsoft.com/solutions/architecture/sharepoint-farm-devtest"
social_image_url: /azure/architecture/solution-ideas/articles/media/sharepoint-farm-devtest.png
name: SharePoint farm for development testing
azureCategories:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This solution addresses the need to deliver a highly available intranet capabili

## Architecture

![Architecture diagram](../media/sharepoint-farm-microsoft-365.png)
![Architecture diagram shows an intranet layout to users on and off premises.](../media/sharepoint-farm-microsoft-365.png)
*Download an [SVG](../media/sharepoint-farm-microsoft-365.svg) of this architecture.*

### Dataflow
Expand Down Expand Up @@ -46,7 +46,7 @@ This solution addresses the need to deliver a highly available intranet capabili
* [Storage Documentation](/azure/storage/blobs/storage-blobs-introduction)
* [Active Directory Documentation](https://support.microsoft.com/help/2721672/microsoft-server-software-support-for-microsoft-azure-virtual-machines)
* [SharePoint Server Documentation](/sharepoint/administration/intranet-sharepoint-server-2016-in-azure-dev-test-environment)
* [SQL Server Documentation](/azure/azure-sql/virtual-machines/windows/create-sql-vm-portal)'
* [SQL Server Documentation](/azure/azure-sql/virtual-machines/windows/create-sql-vm-portal)
* [Load Balancer Documentation](/azure/load-balancer/load-balancer-standard-overview)
* [ExpressRoute Documentation](/azure/expressroute)
* [VPN Gateway Documentation](/azure/vpn-gateway)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ metadata:
titleSuffix: Azure Solution Ideas
description: Deliver highly available intranet capability and share hybrid workloads with Microsoft 365 by using SharePoint servers, Azure Active Directory, and SQL Server.
author: EdPrice-MSFT
ms.author: pnp
ms.date: 12/16/2019
ms.author: edprice
ms.date: 06/14/2022
ms.topic: conceptual
ms.service: architecture-center
ms.subservice: solution-idea
Expand All @@ -22,7 +22,7 @@ metadata:
- sharepoint hybrid solution
- interactive-diagram
- e2e-hybrid
- "https://azure.microsoft.com/solutions/architecture/sharepoint-farm-microsoft-365/"
- "https://azure.microsoft.com/solutions/architecture/sharepoint-farm-microsoft-365"
social_image_url: /azure/architecture/solution-ideas/articles/media/sharepoint-farm-microsoft-365.png
name: Hybrid SharePoint farm with Microsoft 365
azureCategories:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ Start simple with the content management system that enables you to easily maint

## Architecture

![Architecture Diagram](../media/simple-branded-website.png)
![Architecture diagram shows browser to Web Apps then Application Insights and Redis Cache then pulling resources from Azure Content Delivery Network.](../media/simple-branded-website.png)
*Download an [SVG](../media/simple-branded-website.svg) of this architecture.*

### Data flow
### Dataflow

1. User accesses Web Apps from Azure App Service in a browser.
1. Application Insights detects issues and analyzes usage for your web apps.
Expand Down
6 changes: 3 additions & 3 deletions docs/solution-ideas/articles/simple-branded-website.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ metadata:
titleSuffix: Azure Solution Ideas
description: Quickly build and launch digital campaigns that automatically scale. Manage website messaging in real time from a browser with no coding required.
author: EdPrice-MSFT
ms.author: pnp
ms.date: 12/16/2019
ms.author: edprice
ms.date: 06/14/2022
ms.topic: conceptual
ms.service: architecture-center
ms.subservice: solution-idea
Expand All @@ -18,7 +18,7 @@ metadata:
- easy deploy
- content delivery
- interactive-diagram
- "https://azure.microsoft.com/solutions/architecture/simple-branded-website/"
- "https://azure.microsoft.com/solutions/architecture/simple-branded-website"
- internal-intro
social_image_url: /azure/architecture/solution-ideas/articles/media/simple-branded-website.png
name: Simple branded website
Expand Down

0 comments on commit cbfc21e

Please sign in to comment.