New-AzureRmResourceGroupDeployment: loop dependency

[akuryan]

Is there a way to add a dependendency in a loop? I am trying to add several allow all rules to NSG via loop and it fails. I am using such a template:

 {
            "copy": { 
                    "name": "allowCopy", 
                    "count": "[length(parameters('allowedCIDRs'))]" 
                    },  
            "apiVersion": "[variables('apiVersionString')]",
            "type": "Microsoft.Network/networkSecurityGroups/securityRules",
            "name": "[concat(parameters('networkSecurityGroupName'), '/', parameters('allowedCIDRsNames')[copyIndex()])]",
            "location": "[resourceGroup().location]",
              "dependsOn": [
                "[concat('Microsoft.Network/networkSecurityGroups/', parameters('networkSecurityGroupName'))]"
              ],
                "properties": {                     
                    "description": "[concat('Allow everything from ', parameters('allowedCIDRsNames')[copyIndex()])]",
                    "protocol": "*",
                    "sourcePortRange": "*",
                    "destinationPortRange": "*",
                    "sourceAddressPrefix": "[parameters('allowedCIDRs')[copyIndex()]]",
                    "destinationAddressPrefix": "*",
                    "access": "Allow",
                    "priority": "[copyIndex(100)]",
                    "direction": "Inbound"
                }

    }

where allowedCIDRs and allowedCIDRsNames are arrays of 9 elements each.

And it fails with following error:
New-AzureResourceGroupDeployment : 13:55:12 PM - Resource Microsoft.Network/networkSecurityGroups/securityRules 'NSGName/RuleName' failed with message 'Another operation on this or dependent resource is in progress. Toretrieve status of the operation use uri: '

Each time on different rules

[markcowl]

@vivsriaus Any ideas?

[yangl900]

it appears the security rules does not allow concurrent update (copies will update the rule in concurrent by default). We will follow up with owner of this API.

An workaround is to set a dependency on the previous rule, so it does not run in parallel.
{
// provision rule[0]
},
{
"copy": {}
"name": "[copyIndex(1)]" //offset 1 so start with second in the array
"dependsOn": ".../rules/copyIndex()" //set dependency on previous rule
}

[akuryan]

This solution works, but requires a lot of time to provision though :(
My Powershell session ends, and Get-AzureRmResourceGroupDeployment is confirming that deployment took around 20 minutes.
Here is samples I used (just in case they will be useful for someone):
Template itself - https://github.com/akuryan/ConfigurationHelpers/blob/master/Azure/ArmTemplates/nsg-provisioning.json
Some explanation - https://github.com/akuryan/ConfigurationHelpers/blob/master/Azure/ArmTemplates/nsg-provisioning.md

[ravbhatnagar]

@markjbrown - Any ideas on this issue?

[akuryan]

Also, I shall note, that provisioning ends with error in powershell console, but in portal it demonstrates that deployment was ok

[ravbhatnagar]

This should be fixed now. please let us know if you still run into this issue.

[nik-shornikov]

i don't believe this issue is about a specific resource type and i don't believe it's fixed. my experience has been a max of four concurrent subnet writes to a vnet, and anything beyond that, the error above. the hack to force serial does lead to very long deployments, and is a hack to begin with. if arm templates have a value proposition of being declarative, then working out degrees of parallelism and/or observing "retryable" is something this api should learn to do.

Multiple instances of Microsoft.Network/virtualNetworks/subnets/write reliably produce:

{
  "error": {
    "code": "AnotherOperationInProgress",
    "message": "Another operation on this or dependent resource is in progress. To retrieve status of the operation use uri: https://management.azure.com/subscriptions/*/providers/Microsoft.Network/locations/centralus/operations/*?api-version=2016-06-01.",
    "details": []
  }
}

[yangl900]

@nik-shornikov The issue is mostly seen in network resources, it is specific to the resource provider. @ravbhatnagar could you loop the network folks see if there is plan to fix this?

[ravbhatnagar]

@FrankSiegemund - Can you please take a look?

[laughtonsm]

I too am still seeing this issue when trying to perform a similar operation. I'm creating several instances of type Microsoft.Network/networkSecurityGroups/securityRules, which are dependent on a single instance of type Microsoft.Network/networkSecurityGroups. The returned error is as follows:

Template deployment returned the following errors: Resource Microsoft.Network/networkSecurityGroups/securityRules 'nsgazuDEVsvly01/80.80.80.80' failed with message '{ "error": { "code": "AnotherOperationInProgress", "message": "Another operation on this or dependent resource is in progress. To retrieve status of the operation use uri: https://management.azure.com/subscriptions/d76012fa-f8c1-480e-94bf-ffb9ce0e0321/providers/Microsoft.Network/locations/northeurope/operations/5979d8a6-2bdc-40ec-820f-fead1153592d?api-version=2015-06-15.", "details": [] } }' Template output evaluation skipped: at least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.

[jensmh]

I'm having the same issue trying to create multiple subnets in an ARM template copy loop- randomly fails on one or more subnets with the "AnotherOpertionInProgress" error message. Looks like there is a maximum concurency of around 4 writes, as mentioned by @nik-shornikov above.

[ravbhatnagar]

@jensmh @laughtonsm @nik-shornikov - While we try to get the right fix for this issue, may be you can try the serial copies support in ARM as documented here -
https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-multiple#serial-copy

Sample - https://github.com/rjmax/Build2017/blob/master/Act1.TemplateEnhancements/Chapter03.SerialCopies.json

@markjbrown to comment on the original issue around parallelism.