Help with the "right" way to build idempotent Bicep files for resources with network intents...

[abrennan24]

Recently I've been trying to add Azure Data Explorer into my deployment. It requires an existing virtual network, a network security group, and a route table. Subnet delegation is enabled on the subnet which uses network intents to apply the appropriate NSG rules and add the appropriate routes to the route table. This is all works great on the first deployment, but I'm really struggling with how I can make this idempotent. It always fails on the second deployment with conflicts for the NSG and route table.

I'm just trying to define an empty route table and let subnet delegation do its thing:

resource routeTable 'Microsoft.Network/routeTables@2021-05-01' = {
name: 'ADX-route-table'
location: location
properties: {
disableBgpRoutePropagation: false
}
}

I'm not trying to define any rules, but on the second deployment, I get dozens of these errors:

Route Table doesn't have supporting Route for Network Intent Policy Route: Name: 52_224_146_56_32, Id: /subscriptions//resourceGroups//providers/Microsoft.Network/networkIntentPolicies/Microsoft_Kusto_clusters-adx-subnet/routes/52_224_146_56_32, AddressPrefix: 52.224.146.56/32, NextHopType: Internet, NextHopIpAddress: \\\\r\\\\n ----\\\\r\\\\n

I've seen similar discussions (regarding Databricks) advising people to just add all of those rules into their templates. But that seems to me to defeat the whole purpose of delegation. The ADX team could change their NSG rules or any of the IP addresses in the route table at any time. I don't think I want to be adding all that into my template.

The issue with the NSG is essentially the same. I'm trying to define an empty NSG and then define my own rules (about a dozen proxy IP addresses that I need to allow) as separate Microsoft.Network/networkSecurityGroups/securityRules resources.

resource adxNsg 'Microsoft.Network/networkSecurityGroups@2021-05-01' = {
name: 'ADX-nsg'
location: location
properties: {
}
}

Errors:
Network Security Group cannot have resources which conflict with its subnets' network intent policies.\\\\r\\\\n
Network Security Group: /subscriptions//resourceGroups//providers/Microsoft.Network/networkSecurityGroups/ADX-nsg conflicts with Network Intent Policy: Microsoft_Kusto_clusters-adx-subnet\\\\r\\\\n
Network Security Group doesn't have supporting Security Rule for Network Intent Policy Security Rule: Name: AllowAzureDataExplorerManagement, Id: /subscriptions//resourceGroups//providers/Microsoft.Network/networkIntentPolicies/Microsoft_Kusto_clusters-adx-subnet/securityRules/AllowAzureDataExplorerManagement, Access: Allow, Direction: Inbound, Protocol: Tcp, SourceAddressPrefix: AzureDataExplorerManagement, SourcePortRange: 0-65535, DestinationAddressPrefix: VirtualNetwork, DestinationPortRange: 443

I'm sure I could work around this by adding a flag to the deployment if it's the first deployment, and then reference existing resources if it's not. I'm just curious how other people are getting this to work. Is there a better way to handle the subnet delegation / network intents?

[brwilkinson]

I am not familiar with this scenario. It seems like this is something that will not work well with Bicep declarative deployments.

• What are you attaching the NSG's to? Are they specifically required on the subnet or could they be placed on a NIC etc?

A similar scenario is with Just In Time (JIT) policies and NSG.

The way that I manage it is as following.

1. Deploy the blank NSG
   • Attach it to the VM NIC.
2. Deploy the JIT Policy that adds rules to that NSG

By doing this it's easy to manage and Automate.

All prefered NSG rules are on the NSG that is attached to the Subnet, so the dedicated NIC NSG is just for the policy. By doing this can you continue to deploy without any conflict.

E.g.
https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/x.vmJITNSG.bicep

// NSG without any rules
param Deployment string
param VM object

resource vmJMP 'Microsoft.Network/networkSecurityGroups@2021-05-01' = {
  name: '${Deployment}-vm${VM.Name}-JITNSG'
  location: resourceGroup().location
}

Not related to your issue, however a later module that runs to create the JIT policy on the NSG.

https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/x.vmJIT.bicep

In this scenario both of these run from the VM provisioning module

e.g.
https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/VM-VM.bicep#L240
https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/VM-VM.bicep#L343

[brwilkinson]

Other than deploying with no rules, the only other way around this would be to read the current rules, then union them with any current rules that you want to apply. This does work well for some scenarios. If the above doesn't work for you, I can share some of those examples/samples.

[abrennan24]

In this case, the NSG and route table are both applied to the subnet with subnet delegation enabled:

        name: 'adx-subnet'
        properties: {
          addressPrefix: adxSubnet
          routeTable: {
            id: rtId
          }
          networkSecurityGroup: {
            id: adxNsgId
          }
          delegations: [
            {
              name: 'Microsoft.Kusto/clusters'
              properties: {
                serviceName: 'Microsoft.Kusto/clusters'
              }
            }

I don't 100% understand how the subnet delegation functions, but as I understand it, when it's initially enabled on the subnet, it gets the NSG and route table and applies its rules and routes. From that point on, any change to the NSG or route table is validated against the network intents. So when I attempt to deploy an empty route table, it fails those checks and is disallowed.

For the time being, I have implemented a workaround to just pass a deployment flag for the first deployment to create the resource, otherwise to just reference the existing ones. It's not an elegant solution, but it will get the job done for now. If there are ever any enhancements to make this scenario easier, I would definitely be interested.

[brwilkinson]

if you are not needing to deploy your own NSG rules, you could just create a Blank NSG (without rules) and continuously deploy that, however I don't think that would be an option for the route table, so I think your feature flag is the best option.

[brwilkinson]

@abrennan24 going to mark this discussion as complete, since we don't know a fix for this, only the suggested workaround.

[D-Bissell]

Just in case it helps anyone in future:
I had this same issue, but with SQL Managed Instances, which also require delegation on the subnet. Deployments would fail as the NSG and RT I was defining did not have the SQLMI delegated routes/rules, and would violate the network intent policy as they were not defined in the bicep code..

I've seen similar discussions (regarding Databricks) advising people to just add all of those rules into their templates. But that seems to me to defeat the whole purpose of delegation. The ADX team could change their NSG rules or any of the IP addresses in the route table at any time. I don't think I want to be adding all that into my template.

I totally agree.

I managed to solve this (to a degree) by:

1. Getting a reference to the existing NSG/RT
2. Filtering out the delegated routes/rules
3. Union-ing them with the routes/rules I wanted to exist
4. Defining the NSG/RT with the combined routes/rules.

This has worked - deployments are now successful, and any changes I make get realized.

For the route table, for example:

// Network.bicep

// Get reference to the existing route table
resource ref_existingRouteTable 'Microsoft.Network/routeTables@2022-11-01' existing = {
  name: routeTableName
  scope: resourceGroup(resourceGroupName)
}

// Filter existing routes to get only the routes added by SQL MI subnet delegation.
var sqlDelegatedRoutes = filter(ref_existingRouteTable.properties.routes, route => contains(route.name, 'Microsoft.Sql-'))

// Define custom routes
var customRoutes = [
  {
    // Set Default route to go through virtual appliance
    // Overrides Azure system default routes that sends default traffic to nextHopType: Internet
    name: 'DefaultRoute'
    properties: {
      addressPrefix: '0.0.0.0/0'
      nextHopType: 'VirtualAppliance'
      nextHopIpAddress: VirtualApplianceIpAddress
    }
  }
]


// Combine custom routes with existing sql routes, to get the routes to deploy
var routesToDeploy = union(sqlDelegatedRoutes, customRoutes)

// Create Route Table using library resource module
module mod_routeTable '..\library\resources\network\routeTable.resource.bicep' = {
  name: 'mod_RouteTable'
  scope: resourceGroup(resourceGroupName)
  params: {
    name: routeTableName
    location: location
    routes: routesToDeploy
    tags: tags
  }
}


// routeTable.resource.bicep

resource res_routeTable  'Microsoft.Network/routeTables@2022-07-01' = {
  name: name
  location: location
  properties: {
    disableBgpRoutePropagation: disableBgpRoutePropagation
    routes: routes
  }
  tags: tags
}

Notes/Limitations

• Routes had to be defined as properties of the route table resources, not as child resources. Defining them as child resources would make the deployment fail, as it would attempt to remove all the routes first before reapplying them, and again violating network intent policy.
• I think this would fail on an initial deployment, as it fails to find the 'existing' resources that this deployment will be creating.
• The preview/whatif function lists the NSG/RT as 'Ignore', even though the deployment can still make changes to it. Presumably this is due to it using a reference to an existing resource.