resource virtualNetworks with subnets wants to delete the existing subnet

[santo2]

Bicep version
0.4.613 (latest)

Describe the bug
When setting up a template for an existing virtualNetwork resource with subnets as a child resource and a virtualNetworkGateway, deployment wants to delete my existing subnets

• when adding subnets as a separate resource definition (not in the subnets array), it doesn't seem to recognize this and thinks I want to deploy a vnet without subnets. On deployment I get this message:

"message": "Subnet GatewaySubnet is in use by /subscriptions/XXXX/resourceGroups/XXX-TestRelease/providers/Microsoft.Network/virtualNetworkGateways/VPN-C2C-TestRelease/ipConfigurations/vnetGatewayConfig and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet.",

• to be able to reference this subnet in the virtualnetworkgateway, without having to use the "resourceId('')" function, I see only the option to specify the resource as a separate rsource with the parent property, or am I missing an option here? I didn't find any other way.
• since this wasn't possible, I found the workaround to define the subnets in the 'subnets' property of virtualNetworks, and then using the resourceId('') function, although I kind of wished that I didn't need this one.

Template

resource vnet 'Microsoft.Network/virtualNetworks@2021-02-01' = {
  name: vnetName
  location: resourceGroup().location
  properties: {
    addressSpace: {
      addressPrefixes: [
        vnetAddressPefix
      ]
    }
    subnets:[
      {
        name: subnetName
          properties: {
            addressPrefix: vnetSubnetPrefix
            privateEndpointNetworkPolicies: 'Enabled'
            privateLinkServiceNetworkPolicies: 'Enabled'
          }
        }
    ]
    dhcpOptions: {
      dnsServers: vNetDnsServers
    }
  }
}

resource vnetGatewaysubnet  'Microsoft.Network/virtualNetworks/subnets@2021-02-01' =  {
    parent: vnet
    name: 'GatewaySubnet'
    properties: {
      addressPrefix: vnetGatewaySubnetPrefix
      privateEndpointNetworkPolicies: 'Enabled'
      privateLinkServiceNetworkPolicies: 'Enabled'
    }
}

resource virtualNetworkGateway 'Microsoft.Network/virtualNetworkGateways@2021-02-01' = {
  name: gatewayName
  location: resourceGroup().location
  properties: {
    ipConfigurations: [
      {
        name: 'vnetGatewayConfig'
        properties: {
          privateIPAllocationMethod: 'Dynamic'
          subnet: {
            id: vnetGatewaysubnet.id
          }
          publicIPAddress: {
            id: publicIP.id
          }
        }
      }
    ]
    sku: {
      name: gatewaySku
      tier: gatewaySku
    }
    vpnClientConfiguration: {
      vpnClientAddressPool: {
        addressPrefixes: [
          PointToSiteAddressSpace
        ]
      }
      vpnClientProtocols: [
        'SSTP'
      ]
    }
    gatewayType: 'Vpn'
    vpnType: 'RouteBased'
    enableBgp: false
  }
  dependsOn: [
    vnet
  ]
}

To Reproduce

Additional context

[alex-frankel]

This will happen when the subnet is declared as a child resource. It is instead recommended to create all the subnets in the array property inside of the vnet, like you are doing with the first subnet in the above code sample:

resource vnet 'Microsoft.Network/virtualNetworks@2021-02-01' = {
  name: vnetName
  location: resourceGroup().location
  properties: {
    subnets: [
      {
        name: subnetName
        properties: {
          ...
        }
      }
      {
        name: 'GatewaySubnet'
        properties: {
          ...
        }
      }
    ]
  }
  ...
}

This issue is discussed in great length here: Azure/azure-quickstart-templates#2786

We are working with the networking team to update their API to eliminate this problem.

[santo2]

being discussed since 2016, so my hopes are gone... however, I still feel it's very strange that, when mentioned in the same file, it does not "SEE" the defined subnet, and thinks it should be deleted.
Even if I look at your template, it seems rather logical that this behavior should already be in place.
I can only hope my words help convince you to pick this up sooner :-) this prevents me from using the new strenghts of BICEP and the easy way of referencing things by using the actual vnet reference instead of having to search for the a resourceId

[brwilkinson]

Hi @santo2

I am just working on removing the pattern that you ran into with the standalone subnets from the quickstarts to mitigate running into this scenario.

There are 3 patterns that we have at the moment . . . and there is a proposal in place to simplify this with a new 4th option.

1. the simplest and most ARM/Json like method.

resource vnet 'Microsoft.Network/virtualNetworks@2020-06-01' = {
  name: 'vnet1'
  location: resourceGroup().location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/16'
      ]
    }
    subnets: [
      {
        name: 'subnet1'
        properties: {
          addressPrefix: '10.0.0.1/24'
        }
      }
    ]
  }
}

resource networkInterface 'Microsoft.Network/networkInterfaces@2020-11-01' = {
  name: 'name'
  location: resourceGroup().location
  properties: {
    ipConfigurations: [
      {
        name: 'name'
        properties: {
          privateIPAllocationMethod: 'Dynamic'
          subnet: {
            id: resourceId('Microsoft.Network/virtualNetworks/subnets', vnet.name, 'subnet1') // pattern 1 full resourceid lookup
          }                                                // must use the vnet resource (vnet.name) here to generate dependsOn
        }
      }
    ]
  }
}

2. a more Bicep way, however perhaps not ideal

resource vnet 'Microsoft.Network/virtualNetworks@2020-06-01' = {
  name: 'vnet1'
  location: resourceGroup().location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/16'
      ]
    }
    subnets: [
      {
        name: 'subnet1'
        properties: {
          addressPrefix: '10.0.0.1/24'
        }
      }
    ]
  }
}

// standalone existing pattern 2
resource subnet1 'Microsoft.Network/virtualNetworks/subnets@2021-02-01' existing = {
    name: 'subnet1'
    parent: vnet
}
// standalone

resource networkInterface 'Microsoft.Network/networkInterfaces@2020-11-01' = {
  name: 'name'
  location: resourceGroup().location
  properties: {
    ipConfigurations: [
      {
        name: 'name'
        properties: {
          privateIPAllocationMethod: 'Dynamic'
          subnet: {
            id: subnet1.id  // pattern 2 reference for property e.g. id
          }
        }
      }
    ]
  }
}

3. a more Bicep way, however perhaps not ideal

resource vnet 'Microsoft.Network/virtualNetworks@2020-06-01' = {
  name: 'vnet1'
  location: resourceGroup().location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/16'
      ]
    }
    subnets: [
      {
        name: 'subnet1'
        properties: {
          addressPrefix: '10.0.0.1/24'
        }
      }
    ]
  }

  // pattern 3 extra resource block inside of parent
  resource subnet1 'subnets' existing = {
    name: 'subnet1'
  }
} // inside of vnet resource

resource networkInterface 'Microsoft.Network/networkInterfaces@2020-11-01' = {
  name: 'name'
  location: resourceGroup().location
  properties: {
    ipConfigurations: [
      {
        name: 'name'
        properties: {
          privateIPAllocationMethod: 'Dynamic'
          subnet: {
            id: vnet::subnet1.id  // pattern 3 reference for property e.g. id
          }
        }
      }
    ]
  }
}

4. the proposal for a simpler way to reference the subnet/child property

Azure/bicep#2245
Azure/bicep#2246

Below is from 2245

[santo2]

Thanks for the extended answer.
So my assumption that a subnet is a child resource of a vnet and could be made as a child resource, is not correct?
because I seem to miss that option in the ones you state? That seemed for me the easiest and most logical way to define it for re-use.

I followed the mslearn path and this is the first thing popping in my mind after following the course. It does seem to be this way for other resources, so I'm not sure why subnets should be an exception.

resource vnet = {
   resource subnet = {
   }
}
resource anotherOne = {
    subnetId: subnet.id
}

[brwilkinson]

Working backwards....

Whatever subnets are defined inside of the VNET are the subnets that will be deployed/created.

• Assuming templates will always also be redeployed (many times) and the subnets will get updated, from whatever is in the VNET resource definition at that time.

It's likely that over time you may add or remove subnets from a VNET, so all you have to do is define/maintain the VNET resource definition, along with which ever subnets that you want to deploy in that single place.

It's also likely that you may redeploy the VNET in order to update a Route Table or other such settings on a subnet, since these are common network changes. or add Network Address space Etc.

• Note if a subnet is in use, you cannot delete it.

So now to your question...

Yes, you can deploy a Subnet by itself (it's technically possible), however that should be used as a once off capability. An example maybe a failover test and switching a Route Table, Nat Gateway or a NSG OR a DR exercise or such.

It's not advisable to Add a subnet using that capability. Unless you also go an add it back to the VNET resource definition as well. Because of the reasons stated above. So I would just bypass that standalone deployment, just update the VNET and redeploy the VNET instead.

If you do (the standalone) you are having to declare the desired state of a Resource in more than 1 place (assuming your templates IaC are checked into source control). This is not a best practice, since you get configuration drift and a configuration item may get updated in 1 of the 2 places, which would lead to problems.

At the end of the day we also need a way to remove subnets. The capability that facilitates that is just removing it from within the VNET definition and deploying. A Subnet is a property of the VNET. The capability was just added at some point to be able to deploy a subnet by itself and also do a Role Assignment at the Subnet level, which kind of makes sense for people joining NIC's to a subnet Etc, so needing access just on an individual subnet.

[brwilkinson]

An additional comment.

When you make a PUT/Write on a VNET, it's possible to write ALL of the VNET settings/properties including all of the subnets at once, since the subnets are a property of the VNET and it's supported to loop through them with a for loop.

e.g.

Subnets in param file
https://github.com/brwilkinson/AzureDeploymentFramework/blob/7368868d960f34a039c0705ca5b854e3080d03b3/ADF/tenants/AOA/ACU1.T5.parameters.json#L326

VNET definition in Bicep template
https://github.com/brwilkinson/AzureDeploymentFramework/blob/7368868d960f34a039c0705ca5b854e3080d03b3/ADF/bicep/VNET.bicep#L120

When you use the alternate syntax that you mentioned, a standalone PUT of a single subnet on the VNET, you can only write a single subnet that way. Everything else is locked to writes while the single subnet is written/updated.

There are workarounds, however it can be problematic to understand chaining dependOn values as you process each subnet OR setting a batchsize of 1 while it processes these.

These are ALL reasons why the 4 samples above do not include a pattern of writing subnets as if they were child resources.

[ChristopherGLewis]

I've got sample code to show how you do this given the current limitations: vNetBicep. The code in vNet2.Bicep does the entire vNet/Subnet in one pass and eliminates any of the drop/add issues with subnet child objects.

[jikuja]

I've got sample code to show how you do this given the current limitations: vNetBicep. The code in vNet2.Bicep does the entire vNet/Subnet in one pass and eliminates any of the drop/add issues with subnet child objects.

Does it support re-deploying if subnets are used by e.g. private IP addresses? ref: OP's ipConfigurations block.

[ChristopherGLewis]

I've got sample code to show how you do this given the current limitations: vNetBicep. The code in vNet2.Bicep does the entire vNet/Subnet in one pass and eliminates any of the drop/add issues with subnet child objects.

Does it support re-deploying if subnets are used by e.g. private IP addresses? ref: OP's ipConfigurations block.

It should, as long as you get the vnet/Subnet object correct.

I know it will eliminate the Subnet xxx is in use by /subscriptions/XXXX/resourceGroups/YYY error since it doesn't delete the subnet. There are two templates in the repo. Vnet.Bicep uses two json arrays to handle the route table/nsg ID issue. The vNet2.Bicep uses a single json array.

The json array is the hard part - getting this right is pretty much the only way this works. I believe I've noted in the readme that you can use it to add/delete subnets and even change some CIDR ranges, but the ARM engine is pretty restrictive on what's allowed.

[smokedlinq]

I need to be able to do this flow ...

1. deploy vnet
2. deploy subnet for gateway
3. deploy subnet for firewall
4. deploy route table that uses firewall from 3
5. deploy subnet for app with route table from 4

Today I can't do this, I either have to move the subnet creation to CLI outside the bicep or do a two-step manual process because the deployment of the bicep will try to delete the subnets, then the subnet definitions later on will try to put them back. The deployment works until a resource is using an ip from one of the subnets.

[brwilkinson]

@smokedlinq
What is stopping you from being able to define/deploy all of your subnets at the same time as deploying the vnet?

steps:

1. deploy route table (it's a standalone resource)
2. deploy nsg's (it's a standalone resource)
3. deploy vnet with all known subnets defined and route table + nsg's attached.
4. Deploy the firewall
5. Deploy other resources into the App subnet.

Address space and subnet size will need to be determined ahead of time, since you cannot overlap then.

Here is a sample module that has all of the sub modules with dependencies explicitly defined/included for the different stages.

https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/01-ALL-RG.bicep

[smokedlinq]

route table has a dependency on the firewall I missed on step between 3 and 4, deploy azure firewall, that firewall then gets a private ip from the firewall subnet that is used in the route table for the app ...

[brwilkinson]

makes sense, I don't have a Firewall deployed right now, however I am pretty sure that the Firewall internal IP address aks the nexthop address on the route table, similar to what is in the this template below, should be the first usable IP Address in the subnet that you defined. so if the subnet starts at "10.0.1.0/24" it will be "10.0.1.4".

https://docs.microsoft.com/en-us/azure/firewall/quick-create-ipgroup-template

e.g.

  "variables": {
    "vnetAddressPrefix": "10.0.0.0/16",
    "serversSubnetPrefix": "10.0.2.0/24",
    "azureFirewallSubnetPrefix": "10.0.1.0/24",   // <<-- Firewall subnet, first usable is the nextHopIP to define on RT.
    "jumpboxSubnetPrefix": "10.0.0.0/24",
    "nextHopIP": "10.0.1.4",

Basically the first 3 ip's in any Subnet are not usable so if your firewall subnet starts at non-zero, just account for those first three being unavailable.

[smokedlinq]

Ya that's essentially what I've done, guessed at the ip and verified later but I'd rather chain the config together like normal resource linking

[brwilkinson]

Couldn't agree more, that is always my preference.

Even though in this case the IP address is deterministic, so at least it's not a blocker ✅

I am still migrating a few templates from ARM/json to Bicep and i still have some work to do on the Firewall template to address this exact item..

However... so far, in non-Hub environments i.e. Spokes, I opt for the following instead.

• Since the Firewall only exists in the hub and that is deployed prior to spokes.

E.g.

        "RTInfo": [
          {
            "Name": "Hub",
            "Routes": [
              {
                "Name": "Default-Azure-FW",
                "addressPrefix": "0.0.0.0/0",
                "nextHopIpAddress": "FW01",
                "nextHopType": "VirtualAppliance"
              }
            ]
          }
        ],

resource RT 'Microsoft.Network/routeTables@2018-11-01' = [for (RT, i) in RTInfo: {
  name: '${replace(HubVNName, 'vn', 'rt')}${Domain}${RT.Name}'
  location: resourceGroup().location
  properties: {
    routes: [for rt in RT.Routes : {
      name: '${Prefix}-${rt.Name}'
      properties: {
        addressPrefix: rt.addressPrefix
        nextHopType: rt.nextHopType
        nextHopIpAddress: reference(resourceId('Microsoft.Network/azureFirewalls', '${Deployment}-vn${rt.nextHopIpAddress}'), '2021-05-01').ipConfigurations[0].properties.privateIPAddress
      }
    }]
  }
}]

instead of just replying on a naming standard like I do above/below, you would just use the full name of the Firewall as the next hop and do a lookup like that. However again you would need to have a conditional on this between a hub deployment and a spoke deployment.

RT definition:
"nextHopIpAddress": "FW01",

RT template:
'${Deployment}-vn${rt.nextHopIpAddress}

[melugoyal]

What is stopping you from being able to define/deploy all of your subnets at the same time as deploying the vnet?

it appears that parallel subnet creation is not supported, and the only way there appears to be to serialize subnet creation is to separate out the subnet resources from the vnet resource and leverage dependsOn. but then that leads to this problem of the subnets being deleted upon an update of the Bicep template, since the defined vnet resource technically has no defined subnets. i'm not really sure how to circumvent both issues

[brwilkinson]

@melugoyal

Please don't create your subnets individually in a loop, create them as part of the VNET itself.

https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/VNET.bicep#L129

In saying that the best way to overcome your issue, in the rare case you cannot do it in the VNET as just mentioned, is to add the decorator as below, instead of using dependsOn ....

@batchSize(1)

Again however, we have MULTIPLES of threads on why to not deploy subnets with the following syntax and to do it in the VNET instead.

resource subnet1 'Microsoft.Network/virtualNetworks/subnets@2021-02-01'

[YoloClin]

This appears reproducible when running a quickstart guide bicep - https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-bicep?tabs=PowerShell - I ran this bicep file, then without modifications ran it again and get a failure on the second execution.

I'm new to bicep, can somebody demonstrate how the sample code should be fixed?

[brwilkinson]

@YoloClin

you can try to modify the template as follows:

resource virtualNetwork 'Microsoft.Network/virtualNetworks@2021-05-01' = {
  name: virtualNetworkName
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
        addressPrefix
      ]
    }
    subnets: [
      {
        name: subnetName
        properties:{
          addressPrefix: subnetAddressPrefix
          privateEndpointNetworkPolicies: 'Enabled'
          privateLinkServiceNetworkPolicies: 'Enabled'
        }
      }
    ]
  }
}

// resource subnet 'Microsoft.Network/virtualNetworks/subnets@2021-05-01' = {
//   parent: virtualNetwork
//   name: subnetName
//   properties: {
//     addressPrefix: subnetAddressPrefix
//     privateEndpointNetworkPolicies: 'Enabled'
//     privateLinkServiceNetworkPolicies: 'Enabled'
//   }
// }

[YoloClin]

Thanks, looks like I also had to update such that:

resource networkInterface 'Microsoft.Network/networkInterfaces@2021-05-01' = {
  name: networkInterfaceName
  location: location
  properties: {
    ipConfigurations: [
      {
        name: 'ipconfig1'
        properties: {
          subnet: {
            id: virtualNetwork.properties.subnets[0].id
          }

[brwilkinson]

@YoloClin thank you for opening that issue.

[brwilkinson]

Just a note here:

• We have tried to update any quickstarts or examples in Docs, that use the standalone subnet deployment to deploy them within the VNET. If you do come across such cases, please share them, so we can attempt to get them updated.

[Jtango18]

This is a really interesting an informative thread. I have a use case that is currently breaking and would love some guidance (note: also happy to hear that what I'm trying to do is whack!)

I create a VNet as a shared resource. It initially deploys a couple of subnets that will be relevant to parts of my app.

Each part of the app is deployed separately (I'm avoiding using the "micro********" word). Part of that deployment is that they may provision their own subnets for use, given the shared VNet (hub/spoke type model).

This all works wonderfully, right up until I want to change the shared infrastructure with the shared Vnet at which point it balks at some of my spoke subnets and tries to delete them.

What's the best way to work around this? Any guidance much appreciated.

[martinitus]

How about a warning emitted by bicep, when it encounters the nested resource subnets? That might save a dev a day or two of confusion until he eventually stumbles over this issue...

[andrew-sumner]

I'm honestly thinking of going back to terraform. I've hit so many issues trying to create idempotent deployments of anything remotely complex.

[alex-frankel]

This one is admittedly painful and @martinitus suggestion is a good one, but as I understand the networking team is reasonably close to fixing this one properly such that you can declare subnets as child resources again.

I'm going to transfer this to bicep-types-az and link to the larger quickstarts issue thread on the same topic:
Azure/azure-quickstart-templates#2786

[alex-frankel]

I'm honestly thinking of going back to terraform. I've hit so many issues trying to create idempotent deployments of anything remotely complex.

@andrew-sumner - I apologize about the challenges you are running into. Are there other specific cases besides subnets that you want to capture? We want customers to be happy with whatever tool they choose. Terraform is fully supported, so if that is a better tool for you, that is a great option to go with.

[andrew-sumner]

@andrew-sumner - I apologize about the challenges you are running into. Are there other specific cases besides subnets that you want to capture? We want customers to be happy with whatever tool they choose. Terraform is fully supported, so if that is a better tool for you, that is a great option to go with.

Sorry said in a fit of frustration after wasting a day trying to get sql managed instance template re-runnable. Bicep doesn't seem to handle the extra properties / resources it sets / creates very well. Final straw was when vnet resource tried to delete subnets, stumbled across this thread and another to see that the issue has been hanging around for a while.

I've found a lot to love about bicep but these are two examples where it lets itself down.

[mluds]

I used method 3 from this comment to fix this. I'm definitely using Terraform next time though, this is ridiculous.

[wterpstra]

I just got off the phone with Microsoft Support regarding this issue, they asked me to comment my issue here as the 3 workarounds commented above did not work for me.

I'm trying to deploy a Container Apps Environment connected to an infra-subnet in a vnet, the infra-subnet also has a SNAT with public IP address attached so all outbound traffic will originate from that static IP address.

I'm getting the same issue as described here, on an empty resource group it works but when redeploying it breaks with this error (IDs redacted):

{
    "status": "Failed",
    "error": {
        "code": "DeploymentFailed",
        "target": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Resources/deployments/deploy-tooling",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
            {
                "code": "InUseSubnetCannotBeDeleted",
                "message": "Subnet infra-subnet is in use by /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Network/virtualNetworks/{vnetName}/subnets/infra-subnet/serviceAssociationLinks/legionservicelink and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet.",
                "details": []
            }
        ]
    }
}

The bicep involved (summarized & redacted, full bicep is available at support):

param name string
param environmentDTAP string
param location string = resourceGroup().location

resource outboundIp 'Microsoft.Network/publicIPAddresses@2023-09-01' = {
  name: 'pip-${environmentDTAP}'
  location: location
  properties: {
    publicIPAllocationMethod: 'Static'
  }
  sku: {
    name: 'Standard'
    tier: 'Regional'
  }
}

resource natGateway 'Microsoft.Network/natGateways@2023-06-01' = { 
  name: 'snat-${environmentDTAP}'
  location: location
  properties: {
    publicIpAddresses: [
      {
        id: outboundIp.id
      }
    ]
  }
  sku: {
    name: 'Standard'
  }
}

resource vnet 'Microsoft.Network/virtualNetworks@2023-09-01' = {
  name: '${name}-vnet-${environmentDTAP}'
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/16'
      ]
    }
    subnets: [
      {
        name: 'infra-subnet'
        properties: {
          addressPrefix: '10.0.0.0/23'
          natGateway: {
            id: natGateway.id
          }
          delegations: [
            {
              name: 'Microsoft.App.environments'
              properties: {
                serviceName: 'Microsoft.App/environments'
              }
            }
          ]
        }
      }
    ]
  }
  
  resource infraSubnet 'subnets' existing = {
    name: 'infra-subnet'
  }
}

resource env 'Microsoft.App/managedEnvironments@2023-08-01-preview' = {
  name: '${name}-cae-${environmentDTAP}'
  tags: resourceGroup().tags
  location: location
  properties: {
    workloadProfiles: [
      {
        name: 'Consumption'
        workloadProfileType: 'consumption'
      }
      {
        name: 'my-workload-profile'
        workloadProfileType: 'D4'
        minimumCount: 0
        maximumCount: 1
      }
    ] 
    vnetConfiguration: {
      infrastructureSubnetId: vnet::infraSubnet.id
      internal: false
    }
  }
}

output id string = env.id
output name string = env.name

[jikuja]

@wterpstra Did they mention that fix is on public preview?

ref: Azure/azure-quickstart-templates#2786 (comment)

[wterpstra]

@jikuja Very interesting, thanks. I don't think we should expect to see this feature in bicep/arm though. Bicep/arm should be idempotent, omitting a value for subnets should result in all subnets of the vnet to be removed, not ignored like in the mentioned blog post.

The vnet in my bicep has a subnet configured, on redeploy the subnet was not changed or removed, so I expect nothing to be changed. Still, it is trying to remove my subnet.

Update: I just read here that it should be working on bicep too. I'll test my bicep in one of the mentioned euap regions

[wterpstra]

Okay, I got it working! I moved all my networking resources into a single bicep module (separate from the container app environment resource). Then pass the name of the vnet to the module containing the container app environment and referencing the vnet and subnet like this (similar to option 3 mentioned above):

resource vnet 'Microsoft.Network/virtualNetworks@2023-09-01' existing =  {
  name: vnetName
  resource infraSubnet 'subnets' existing = {
    name: 'infra-subnet'
  }
}

var subnetId = vnet::infraSubnet.id

[zukakosan]

It seems that this update may be a solution for this problem, isn't it?

https://techcommunity.microsoft.com/t5/azure-networking-blog/azure-virtual-network-now-supports-updates-without-subnet/ba-p/4067952

But if you want to use standalone subnet, all subnets must be declared as standalne. In the VNet part, subnets mustn't exist because other subnets are deleted.

I haven't actually try this update yet, this opinion includes my expectation.

[zukakosan]

I validated and confirmed that the new behavior works good in this context.

I can deploy subnet again with this code (new behavior),
https://github.com/zukakosan/bicep-learn/blob/main/20240319-standalonevnet/main.bicep

while come up with the error with this code including subnet definition in vnet part.
https://github.com/zukakosan/bicep-learn/blob/main/20240319-standalonevnet/main-error.bicep

[celomaluf]

In addition to what @zukakosan shared, the API needs to be on at least version "2023-11-01" for both vnet and subnet resources.
Thanks so much for sharing it, it works like a charm :).

[avivansh]

I tried using both API-Version @2023-11-01 & @2023-09-01. But I am still getting the below error when I try to re-deploy my bicep templates.

Error: -

"details":[{"code":"InUseSubnetCannotBeUpdated","message":"Subnet snet-cosno is in use and cannot be updated."

Context: -
I have my virtual network defined and I am creating 2 subnets snet-cosno & snet-kv. Azure cosmos DB is already provisioned and is in the snet-cosno subnet of the vnet. Now when I re-run the deployment, it results in the above error. I am not sure why it's updating snet-cosno subnet when there has been no changes made to my bicep templates. I was assumed the operation would be idempotent and it won't result into any errors.

Code:-

targetScope = 'resourceGroup'

param name string

param region string

param addressPrefix string

param subnets array

param tags object

param ddosProtectionPlanId string

var subnets = [
  {
    name: 'snet-cosno'
    properties: {
      addressPrefix: '<REDACTED>'
      privateLinkServiceNetworkPolicies: 'Enabled'
      privateEndpointNetworkPolicies: 'Disabled'
      networkSecurityGroup: {
        id: resourceId('<REDACTED>')
      }
    }
  }
  {
    name: 'snet-kv'
    properties: {
      addressPrefix: '<REDACTED>'
      privateLinkServiceNetworkPolicies: 'Enabled'
      privateEndpointNetworkPolicies: 'Disabled'
      networkSecurityGroup: {
        id: resourceId('<REDACTED>')
      }
  }
]

resource virtualNetwork 'Microsoft.Network/virtualNetworks@2023-11-01' = {
  name: name
  location: region
  properties: {
    addressSpace: {
      addressPrefixes: [addressPrefix]
    }
    enableDdosProtection: true
    ddosProtectionPlan: {
      id: ddosProtectionPlanId
    }
    subnets: subnets
  }
  tags: tags
}

output name string = virtualNetwork.name

Any help, why re-deployment isn't idempotent here? why snet-cosno is getting updated even when there isn't any change?

[avivansh]

I validated and confirmed that the new behavior works good in this context.

I can deploy subnet again with this code (new behavior), https://github.com/zukakosan/bicep-learn/blob/main/20240319-standalonevnet/main.bicep

while come up with the error with this code including subnet definition in vnet part. https://github.com/zukakosan/bicep-learn/blob/main/20240319-standalonevnet/main-error.bicep

While running the main.bicep twice, I am getting the error InUseSubnetCannotBeDeleted. I don't think it's idempotent. @zukakosan is this expected?

[{"code":"InUseSubnetCannotBeDeleted","message":"Subnet subnet-vm is in use by /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Network/networkInterfaces/vm-ubuntu-001-nic/ipConfigurations/ipconfig1 and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet.","details":[]}]

[zukakosan]

@avivansh
Thanks for referring my repository. I updated the API version from 2023-09-01 to 2023-11-01. I've just tried to deploy multiple times, in my environment, that code is working after second deployment. The region is eastus2euap.

Please try my new code.

[avivansh]

@zukakosan, I am trying to deploy in eastus region, it doesn't work for me.

[zukakosan]

@avivansh
I tried in eastus just in case, but no errors come up. I think your env has accidentally something wrong.

shizh@bicep-learn/20240319-standalonevnet (main)
$ az deployment group create -g '20240701-eus-2' --template-file main.bicep
Please provide string value for 'adminUsername' (? for help): AzureAdmin
Please provide securestring value for 'adminPassword' (? for help): 
{
  "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Resources/deployments/main",
  "location": null,
  "name": "main",
  "properties": {
    "correlationId": "8dec081a-8a68-41c9-8870-78610ce90251",
    "debugSetting": null,
    "dependencies": [
      {
        "dependsOn": [
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "vnet-test",
            "resourceType": "Microsoft.Network/virtualNetworks"
          },
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkSecurityGroups/nsg-default",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "nsg-default",
            "resourceType": "Microsoft.Network/networkSecurityGroups"
          }
        ],
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-001",
        "resourceGroup": "20240701-eus-2",
        "resourceName": "vnet-test/subnet-001",
        "resourceType": "Microsoft.Network/virtualNetworks/subnets"
      },
      {
        "dependsOn": [
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "vnet-test",
            "resourceType": "Microsoft.Network/virtualNetworks"
          },
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-001",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "vnet-test/subnet-001",
            "resourceType": "Microsoft.Network/virtualNetworks/subnets"
          },
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkSecurityGroups/nsg-default",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "nsg-default",
            "resourceType": "Microsoft.Network/networkSecurityGroups"
          }
        ],
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-vm",
        "resourceGroup": "20240701-eus-2",
        "resourceName": "vnet-test/subnet-vm",
        "resourceType": "Microsoft.Network/virtualNetworks/subnets"
      },
      {
        "dependsOn": [
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-vm",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "vnet-test/subnet-vm",
            "resourceType": "Microsoft.Network/virtualNetworks/subnets"
          }
        ],
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkInterfaces/vm-ubuntu-test-nic",
        "resourceGroup": "20240701-eus-2",
        "resourceName": "vm-ubuntu-test-nic",
        "resourceType": "Microsoft.Network/networkInterfaces"
      },
      {
        "dependsOn": [
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkInterfaces/vm-ubuntu-test-nic",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "vm-ubuntu-test-nic",
            "resourceType": "Microsoft.Network/networkInterfaces"
          }
        ],
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Compute/virtualMachines/vm-ubuntu-test",
        "resourceGroup": "20240701-eus-2",
        "resourceName": "vm-ubuntu-test",
        "resourceType": "Microsoft.Compute/virtualMachines"
      }
    ],
    "duration": "PT1M1.7763697S",
    "error": null,
    "mode": "Incremental",
    "onErrorDeployment": null,
    "outputResources": [
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Compute/virtualMachines/vm-ubuntu-test",
        "resourceGroup": "20240701-eus-2"
      },
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkInterfaces/vm-ubuntu-test-nic",
        "resourceGroup": "20240701-eus-2"
      },
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkSecurityGroups/nsg-default",
        "resourceGroup": "20240701-eus-2"
      },
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test",
        "resourceGroup": "20240701-eus-2"
      },
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-001",
        "resourceGroup": "20240701-eus-2"
      },
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-vm",
        "resourceGroup": "20240701-eus-2"
      }
    ],
    "outputs": null,
    "parameters": {
      "adminPassword": {
        "type": "SecureString"
      },
      "adminUsername": {
        "type": "String",
        "value": "AzureAdmin"
      }
    },
    "parametersLink": null,
    "providers": [
      {
        "id": null,
        "namespace": "Microsoft.Network",
        "providerAuthorizationConsentState": null,
        "registrationPolicy": null,
        "registrationState": null,
        "resourceTypes": [
          {
            "aliases": null,
            "apiProfiles": null,
            "apiVersions": null,
            "capabilities": null,
            "defaultApiVersion": null,
            "locationMappings": null,
            "locations": [
              "eastus"
            ],
            "properties": null,
            "resourceType": "networkSecurityGroups",
            "zoneMappings": null
          },
          {
            "aliases": null,
            "apiProfiles": null,
            "apiVersions": null,
            "capabilities": null,
            "defaultApiVersion": null,
            "locationMappings": null,
            "locations": [
              "eastus"
            ],
            "properties": null,
            "resourceType": "virtualNetworks",
            "zoneMappings": null
          },
          {
            "aliases": null,
            "apiProfiles": null,
            "apiVersions": null,
            "capabilities": null,
            "defaultApiVersion": null,
            "locationMappings": null,
            "locations": [
              null
            ],
            "properties": null,
            "resourceType": "virtualNetworks/subnets",
            "zoneMappings": null
          },
          {
            "aliases": null,
            "apiProfiles": null,
            "apiVersions": null,
            "capabilities": null,
            "defaultApiVersion": null,
            "locationMappings": null,
            "locations": [
              "eastus"
            ],
            "properties": null,
            "resourceType": "networkInterfaces",
            "zoneMappings": null
          }
        ]
      },
      {
        "id": null,
        "namespace": "Microsoft.Compute",
        "providerAuthorizationConsentState": null,
        "registrationPolicy": null,
        "registrationState": null,
        "resourceTypes": [
          {
            "aliases": null,
            "apiProfiles": null,
            "apiVersions": null,
            "capabilities": null,
            "defaultApiVersion": null,
            "locationMappings": null,
            "locations": [
              "eastus"
            ],
            "properties": null,
            "resourceType": "virtualMachines",
            "zoneMappings": null
          }
        ]
      }
    ],
    "provisioningState": "Succeeded",
    "templateHash": "15424642090958167980",
    "templateLink": null,
    "timestamp": "2024-07-01T13:35:31.531748+00:00",
    "validatedResources": null
  },
  "resourceGroup": "20240701-eus-2",
  "tags": null,
  "type": "Microsoft.Resources/deployments"
}

shizh@bicep-learn/20240319-standalonevnet (main)
$ az deployment group create -g '20240701-eus-2' --template-file main.bicep 
Please provide string value for 'adminUsername' (? for help): AzureAdmin
Please provide securestring value for 'adminPassword' (? for help):
{
  "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Resources/deployments/main",
  "location": null,
  "name": "main",
  "properties": {
    "correlationId": "dc99891e-91e4-45fd-bc8a-10781dd6e839",
    "debugSetting": null,
    "dependencies": [
      {
        "dependsOn": [
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "vnet-test",
            "resourceType": "Microsoft.Network/virtualNetworks"
          },
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkSecurityGroups/nsg-default",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "nsg-default",
            "resourceType": "Microsoft.Network/networkSecurityGroups"
          }
        ],
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-001",
        "resourceGroup": "20240701-eus-2",
        "resourceName": "vnet-test/subnet-001",
        "resourceType": "Microsoft.Network/virtualNetworks/subnets"
      },
      {
        "dependsOn": [
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "vnet-test",
            "resourceType": "Microsoft.Network/virtualNetworks"
          },
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-001",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "vnet-test/subnet-001",
            "resourceType": "Microsoft.Network/virtualNetworks/subnets"
          },
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkSecurityGroups/nsg-default",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "nsg-default",
            "resourceType": "Microsoft.Network/networkSecurityGroups"
          }
        ],
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-vm",
        "resourceGroup": "20240701-eus-2",
        "resourceName": "vnet-test/subnet-vm",
        "resourceType": "Microsoft.Network/virtualNetworks/subnets"
      },
      {
        "dependsOn": [
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-vm",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "vnet-test/subnet-vm",
            "resourceType": "Microsoft.Network/virtualNetworks/subnets"
          }
        ],
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkInterfaces/vm-ubuntu-test-nic",
        "resourceGroup": "20240701-eus-2",
        "resourceName": "vm-ubuntu-test-nic",
        "resourceType": "Microsoft.Network/networkInterfaces"
      },
      {
        "dependsOn": [
          {
            "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkInterfaces/vm-ubuntu-test-nic",
            "resourceGroup": "20240701-eus-2",
            "resourceName": "vm-ubuntu-test-nic",
            "resourceType": "Microsoft.Network/networkInterfaces"
          }
        ],
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Compute/virtualMachines/vm-ubuntu-test",
        "resourceGroup": "20240701-eus-2",
        "resourceName": "vm-ubuntu-test",
        "resourceType": "Microsoft.Compute/virtualMachines"
      }
    ],
    "duration": "PT11.161678S",
    "error": null,
    "mode": "Incremental",
    "onErrorDeployment": null,
    "outputResources": [
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Compute/virtualMachines/vm-ubuntu-test",
        "resourceGroup": "20240701-eus-2"
      },
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkInterfaces/vm-ubuntu-test-nic",
        "resourceGroup": "20240701-eus-2"
      },
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/networkSecurityGroups/nsg-default",
        "resourceGroup": "20240701-eus-2"
      },
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test",
        "resourceGroup": "20240701-eus-2"
      },
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-001",
        "resourceGroup": "20240701-eus-2"
      },
      {
        "id": "/subscriptions/xxxx/resourceGroups/20240701-eus-2/providers/Microsoft.Network/virtualNetworks/vnet-test/subnets/subnet-vm",
        "resourceGroup": "20240701-eus-2"
      }
    ],
    "outputs": null,
    "parameters": {
      "adminPassword": {
        "type": "SecureString"
      },
      "adminUsername": {
        "type": "String",
        "value": "AzureAdmin"
      }
    },
    "parametersLink": null,
    "providers": [
      {
        "id": null,
        "namespace": "Microsoft.Network",
        "providerAuthorizationConsentState": null,
        "registrationPolicy": null,
        "registrationState": null,
        "resourceTypes": [
          {
            "aliases": null,
            "apiProfiles": null,
            "apiVersions": null,
            "capabilities": null,
            "defaultApiVersion": null,
            "locationMappings": null,
            "locations": [
              "eastus"
            ],
            "properties": null,
            "resourceType": "networkSecurityGroups",
            "zoneMappings": null
          },
          {
            "aliases": null,
            "apiProfiles": null,
            "apiVersions": null,
            "capabilities": null,
            "defaultApiVersion": null,
            "locationMappings": null,
            "locations": [
              "eastus"
            ],
            "properties": null,
            "resourceType": "virtualNetworks",
            "zoneMappings": null
          },
          {
            "aliases": null,
            "apiProfiles": null,
            "apiVersions": null,
            "capabilities": null,
            "defaultApiVersion": null,
            "locationMappings": null,
            "locations": [
              null
            ],
            "properties": null,
            "resourceType": "virtualNetworks/subnets",
            "zoneMappings": null
          },
          {
            "aliases": null,
            "apiProfiles": null,
            "apiVersions": null,
            "capabilities": null,
            "defaultApiVersion": null,
            "locationMappings": null,
            "locations": [
              "eastus"
            ],
            "properties": null,
            "resourceType": "networkInterfaces",
            "zoneMappings": null
          }
        ]
      },
      {
        "id": null,
        "namespace": "Microsoft.Compute",
        "providerAuthorizationConsentState": null,
        "registrationPolicy": null,
        "registrationState": null,
        "resourceTypes": [
          {
            "aliases": null,
            "apiProfiles": null,
            "apiVersions": null,
            "capabilities": null,
            "defaultApiVersion": null,
            "locationMappings": null,
            "locations": [
              "eastus"
            ],
            "properties": null,
            "resourceType": "virtualMachines",
            "zoneMappings": null
          }
        ]
      }
    ],
    "provisioningState": "Succeeded",
    "templateHash": "15424642090958167980",
    "templateLink": null,
    "timestamp": "2024-07-01T13:36:20.945473+00:00",
    "validatedResources": null
  },
  "resourceGroup": "20240701-eus-2",
  "tags": null,
  "type": "Microsoft.Resources/deployments"
}

[apryiomka]

This will happen when the subnet is declared as a child resource. It is instead recommended to create all the subnets in the array property inside of the vnet, like you are doing with the first subnet in the above code sample:

resource vnet 'Microsoft.Network/virtualNetworks@2021-02-01' = {
  name: vnetName
  location: resourceGroup().location
  properties: {
    subnets: [
      {
        name: subnetName
        properties: {
          ...
        }
      }
      {
        name: 'GatewaySubnet'
        properties: {
          ...
        }
      }
    ]
  }
  ...
}

This issue is discussed in great length here: Azure/azure-quickstart-templates#2786

We are working with the networking team to update their API to eliminate this problem.

This still happens regardless. We define subnets array in the vnet resource, yet if the subnet created outside of bicep, the deployment will try to delete them.