Azure Virtual Machine Access To Blob Storage With User Identity

The developer at Mystique Unicorn are looking for a way store their order events coming from their stores across the world. They are currently using Azure Blob storage for their data lake and would like to store these events there. A sample of the event is shown below.

{
    "request_id": "fef1127f-65df-4bca-bc58-e102fb65ee70",
    "store_id": 6,
    "store_fqdn": "m-web-srv-011.internal.cloudapp.net",
    "store_ip": "10.0.0.4",
    "cust_id": 989,
    "category": "Laptops",
    "sku": 126551,
    "price": 10.61,
    "qty": 6,
    "discount": 2.7,
    "gift_wrap": false,
    "variant": "black",
    "priority_shipping": true,
    "ts": "2023-04-10T16:25:04.411091",
    "contact_me": "github.com/miztiik"
}

They would like to store the data by date so that they can query them easily. For example query the sales across stores based on store id on a given date.

Can you show them how the can get started?

🎯 Solutions

To store all the order events, we will create a blob container. To write the events to the container, lets use a user managed identity with it permission scoped narrowly to a specific blob conainer using Azure RBAC.^{1, 2, 3}.

1. 🧰 Prerequisites

   This demo, instructions, scripts and bicep template is designed to be run in westeurope. With few or no modifications you can try it out in other regions as well(Not covered here).

   • 🛠 Azure CLI Installed & Configured - Get help here
   • 🛠 Bicep Installed & Configured - Get help here
   • 🛠 VS Code & Bicep Extenstions - Get help here

2. ⚙️ Setting up the environment

   • Get the application code

     git clone https://github.com/miztiik/azure-vm-to-blob-storage
     cd azure-vm-to-blob-storage

3. 🚀 Prepare the environment

   Let check you have Azure Cli working with

     # You should have azure cli preinstalled
     az account show

   You should see an output like this,

    {
      "environmentName": "AzureCloud",
      "homeTenantId": "16b30820b6d3",
      "id": "1ac6fdbff37cd9e3",
      "isDefault": true,
      "managedByTenants": [],
      "name": "YOUR-SUBS-NAME",
      "state": "Enabled",
      "tenantId": "16b30820b6d3",
      "user": {
        "name": "miztiik@",
        "type": "user"
      }
    }

4. 🚀 Deploying the application

   Let us walk through each of the stacks,

   • Stack: Main Bicep The params required for the modules are in params.json. Do modify them to suit your need.(Especially the adminPassword.secureString for the VM. You are strongly encouraged to Just-In-Time access or use SSH key instead of password based authentication). The helper deployment script deploy.sh will deploy the main.bicep file. This will create the following resoureces

     • Resource Group(RG)
     • VNet, Subnet & Virtual Machine
     • Virtual Machine(Ubuntu)
       • (WIP)Bootstrapped with custom libs using userData script.
     • Storage Account - warehouseXXXX
       • Blob Container - store-events-xxx
     • User Managed Identity
       • Scoped with contributor privileges with conditional access restricting to a container, (For ex store-events-xxx)
     • Identity attached to the VM

     After successfully deploying the stack, Check the Resource Groups/Deployments section for the resources.

5. 🔬 Testing the solution

   • Connect to the VM

     The Ubuntu vm should be bootstrapped using userData to install python3, git and also Azure Blob Storage Client Library⁴. (_For some reason the userData, bootstrap script fails, Let me know if know how to fix it)

     • Connect to the using using Just In Time Access⁵.

     • Copy the az_producer.py to the server.

     • The python script expects the storage account name (SA_NAME) and blob container name (CONTAINER_NAME) as environment variables. We need to set them before execution.(Update to your values in the below code)

       ssh miztiik@publicIP
       git clone https://github.com/miztiik/azure-vm-to-blob-storage
       cd azure-vm-to-blob-storage
       
       # If pre-reqs have not been installed, run the bootstrap script manually
       # bash /var/azure-vm-to-blob-storage/modules/vm/bootstrap_scripts/deploy_app.sh
       export SA_NAME="warehouse4moddm014"
       export CONTAINER_NAME="store-events-014"
       python3 /var/azure-vm-to-blob-storage/app/az_producer.py

       If everything goes all right, you should see messages like this,

       INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https://warehouse4moddm014.blob.core.windows.net/store-events-014/sales_events/dt%3D2023_04_10/1681153081463131.json'
       Request method: 'PUT'
       Request headers:
           'Content-Length': '386'
           'x-ms-blob-type': 'REDACTED'
           'If-None-Match': '*'
           'x-ms-version': 'REDACTED'
           'Content-Type': 'application/octet-stream'
           'Accept': 'application/xml'
           'Last-Modified': 'Mon, 10 Apr 2023 18:58:01 GMT'
           'ETag': '"0x8DB39F5820D3CEA"'
           'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
           'x-ms-request-id': 'ded43899-001e-0019-62de-6bf3bd000000'
           'x-ms-client-request-id': '9dd7975a-d7d1-11ed-82ed-0bcd26fbfe68'
           'x-ms-version': 'REDACTED'
           'x-ms-content-crc64': 'REDACTED'
           'x-ms-request-server-encrypted': 'REDACTED'
           'Date': 'Mon, 10 Apr 2023 18:58:00 GMT'
       INFO:root:Blob sales_events/dt=2023_04_10/1681153081463131.json uploaded successfully

     The script should create store events and write them to Blob container. The path should be something like this store-events-014/sales_events/dt=2023_04_10/1681153079333515.json

     

6. Troubleshooting Azure Monitor Agent

   • Check if the VM can write to blob using cli. List Blobs

       RG_NAME="MIZTIIK_ENTERPRISES_AZURE_VM_TO_BLOB_STORAGE_011"
       SA_NAME="warehousei5chd4011"
       CONTAINER_NAME="store-events-011"
     
       az storage blob list \
           --container-name ${CONTAINER_NAME1} \
           --account-name ${SA_NAME} \
           --auth-mode login
     
       az storage blob directory list \
           --container-name ${CONTAINER_NAME} \
           -d default \
           --account-name ${SA_NAME} \
           --auth-mode login

     Upload file to blob,

     echo "hello world on $(date +'%Y-%m-%d')" > miztiik.log
     az storage blob upload \
       --account-name ${SA_NAME} \
       --container-name ${CONTAINER_NAME} \
       --name miztiik.log \
       --file miztiik.log \
       --auth-mode login

7. 📒 Conclusion

   Here we have demonstrated how to store files or byte stream data to Azure Blob Containers. You can extend the solution to setup trigger in blob conatiners to further process these events.

8. 🧹 CleanUp

If you want to destroy all the resources created by the stack, Execute the below command to delete the stack, or you can delete the stack from console as well

• Resources created during Deploying The Application
• Any other custom resources, you have created for this demo

# Delete from resource group
az group delete --name Miztiik_Enterprises_xxx --yes
# Follow any on-screen prompt

This is not an exhaustive list, please carry out other necessary steps as maybe applicable to your needs.

📌 Who is using this

This repository aims to show how to Bicep to new developers, Solution Architects & Ops Engineers in Azure.

💡 Help/Suggestions or 🐛 Bugs

Thank you for your interest in contributing to our project. Whether it is a bug report, new feature, correction, or additional documentation or solutions, we greatly value feedback and contributions from our community. Start here

👋 Buy me a coffee

Buy me a coffee ☕.

📚 References

1. Azure Docs: Azure RBAC
2. Azure Docs: Azure ABAC
3. Azure Docs: Azure RBAC Example Conditions
4. Azure Docs: Azure Blob Storage client library for Python
5. Azure Docs: Just In Time Access

🏷️ Metadata

Level: 200