Orchestrating Events with Azure Service Bus Message Filters and Azure Functions using Managed Identity

Mystique Unicorn's developers are in need of guidance to implement event processing for orchestrating events across multiple applications and services, utilizing Azure's native capabilities. They aim to have the ability to filter specific spends based on certain properties. For instance, they want to filter events that are classified as sale_event and have a discount greater than 20. This filtering mechanism would be valuable for analyzing business metrics such as total_sales and total_discounts, as well as detecting potential fraudulent activity.

{
  "id": "743da362-69df-4e63-a95f-a1d93e29825e",
  "request_id": "743da362-69df-4e63-a95f-a1d93e29825e",
  "store_id": 5,
  "store_fqdn": "localhost",
  "store_ip": "127.0.0.1",
  "cust_id": 549,
  "category": "Notebooks",
  "sku": 169008,
  "price": 45.85,
  "qty": 34,
  "discount": 10.3,
  "gift_wrap": true,
  "variant": "red",
  "priority_shipping": false,
  "ts": "2023-05-19T14:36:09.985501",
  "contact_me": "github.com/miztiik",
  "is_return": true
}

Event properties,

{
   "event_type":"sale_event",
   "priority_shipping":false,
   "discount":10
}

Can you provide guidance on how to accomplish this?

🎯 Solution

We can utilize the capabilities of Azure Service Bus Topic to filter messages based on message properties. Currently, there is no capability to filter messages based on the message body itself. Therefore, any attribute required for filtering needs to be included in the message property.

In this demonstration, an Azure Function with a managed identity will generate events and send them to a topic within a designated service bus namespace. To support the filtering process, 4 subscriptions have been created for this topic.

• All Events : Uses the SQL Filter 1=1
• Inventory_Events : Uses the Correlation filter to find events of type inventory_event
• Sale_Events: Uses the Correlation filter to find events of type sale_event
• Fraud_Events: Uses the SQL Filter event_type = 'sale_event' AND discount > 20

The consumer function utilizes a subscription trigger on the sale_events topic to consume events from the topic. It then processes and persists these events to an Azure Storage Account and Cosmos DB. To ensure secure and controlled access to the necessary resources, a scoped managed identity with RBAC (Role-Based Access Control) permissions is employed. Additionally, the trigger connection is authenticated using a managed identity.

By leveraging the capabilities of Bicep, all the required resources can be easily provisioned and managed with minimal effort. Bicep simplifies the process of defining and deploying Azure resources, allowing for efficient resource management.

1. 🧰 Prerequisites

   This demo, instructions, scripts and bicep template is designed to be run in westeurope. With few or no modifications you can try it out in other regions as well(Not covered here).

   • 🛠 Azure CLI Installed & Configured - Get help here
   • 🛠 Bicep Installed & Configured - Get help here
   • 🛠 VS Code & Bicep Extenstions - Get help here

2. ⚙️ Setting up the environment

   • Get the application code

     https://github.com/miztiik/azure-svc-bus-msg-consumer-with-filter
     cd azure-svc-bus-msg-consumer-with-filter

3. 🚀 Prepare the environment

   Ensure you have Azure Cli and bicep working

   # You should have azure cli preinstalled
   bicep --version
   az account show

4. 🚀 Deploying the application

   • Stack: Main Bicep We will create the following resoureces

     • Storage Accounts for storing the events
       • General purpose Storage Account - Used by Azure functions to store the function code
       • warehouse** - Azure Function will store the events here
     • Servie Bus Namespace
       • Service Bus Topic, with following subscriptions
         • All Events
         • Inventory_Events
         • Sale_Events
         • Fraud_Events
     • Managed Identity
       • This will be used by the Azure Function to interact with the service bus
     • Python Azure Function
       • Producer: HTTP Trigger. Customized to send count number of events to the service bus, using parameters passed in the query string. count defaults to 10
       • Consumer: Service Bus Topic Subscription Trigger. The trigger uses managed identity to authenticate to the service bus.
     • Cosmos DB
       • This will be used by the Azure Function to store the events

     # make deploy
     sh deployment_scripts/deploy.sh

     After successfully deploying the stack, Check the Resource Groups/Deployments section for the resources.

5. 🔬 Testing the solution

   • Trigger the function

     FUNC_URL="https://svc-bus-msg-filter-store-backend-fn-app-004.azurewebsites.net/api/store-events-producer-fn"
     curl ${FUNC_URL}?count=10

     You should see an output like this,

        {
        "miztiik_event_processed": true,
        "msg": "Generated 10 messages",
        "resp": {
           "status": true,
           "tot_msgs": 10,
           "bad_msgs": 2,
           "sale_evnts": 8,
           "inventory_evnts": 2,
           "tot_sales": 507.57
        },
        "count": 10,
        "last_processed_on": "2023-05-21T14:03:45.505416"
        }
     

     Our function has produced 10 messages with 8 of them were sale_events and 2 of them were inventory_events. We can find the same being processed by sale_events and inventory_events subscriptions. Likewise, we see 8 records in blob and cosmos.

     

6. 📒 Conclusion

   In this demonstration, we have showcased the utilization of Azure Functions to send messages to Azure Service Bus and leverage message filters to perform actions on specific messages. This approach allows for efficient message processing and enables targeted handling based on specific criteria.

7. 🧹 CleanUp

   If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well

   • Resources created during Deploying The Application
   • Any other custom resources, you have created for this demo

   # Delete from resource group
   az group delete --name Miztiik_Enterprises_xxx --yes
   # Follow any on-screen prompt

   This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.

📌 Who is using this

This repository aims to show how to Bicep to new developers, Solution Architects & Ops Engineers in Azure.

💡 Help/Suggestions or 🐛 Bugs

Thank you for your interest in contributing to our project. Whether it is a bug report, new feature, correction, or additional documentation or solutions, we greatly value feedback and contributions from our community. Start here

👋 Buy me a coffee

Buy me a coffee ☕.

📚 References

1. Azure Docs - Service Bus Topic Filters
2. Azure Docs - Service Bus Topic Filter Limitations
3. Azure Docs - Managed Identity
4. Azure Docs - Managed Identity Caching
5. Gitub Issue - Default Credential Troubleshooting
6. Gitub Issue - Default Credential Troubleshooting

🏷️ Metadata

Level: 200