Bicep/arm deployment fail after configuring Front Door managed certificate with failure message: "Operations on Secret Type ManagedCertificate are not allowed."

[aviita]

Bicep version
Bicep CLI version 0.7.4 (5afc312)

Describe the bug
I have been setting up Azure Front Door manually, exporting ARM template and converting it to bicep with az bicep decompile, then formatting it to more generic script. All went well up-to a point where I added custom domains. We are using external DNS service so I saw no possibilities to automate the custom domain setup and instead went with manual setup first.

I ran into two problems with bicep. Firstly I did not find a way to make the bicep deploy ignore the DNS configurations after this manual setup. And secondly after trying to add the custom domains to bicep script by again exporting ARM template, converting to bicep and modifying my bicep script with similar changes as were added by bicep, the az deploy starts to fail with error message.

To Reproduce
Steps to reproduce the behavior:

1. Manually configure Azure front door with endpoint and route pointing to Azure App Service plan
2. Export ARM template json
3. Convert ARM template to bicep with az bicep decompile
4. Configure custom domain

• Do all the external DNS verification settings in this step, inluding TXT record for _dnsauth.subdomain.mydomain.com, CNAME with afdverify.subdomain and CNAME with subdomain to finally route to front door endpoin

5. Export ARM template json
6. Convert ARM template to bicep with az bicep decompile
7. Deploy Front Door with ARM template exported on step 5.

Expected

• az deploy succeeds with bicep script that has custom domain in place
• az deploy succeeds with ARM template exported after custom domain setup that has seen no changes

Actual

• az deploy fails with bicep script that has custom domain in place
• az deploy fails with ARM template exported after custom domain setup that has seen no changes

az deploy error message when deploying with ARM template after step 5 or bicep with script after step 6.

{"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"MethodNotAllowed","message":"{\r\n  \"error\": {\r\n    \"code\": \"OperationNotAllowed\",\r\n    \"message\": \"Operations on Secret Type ManagedCertificate are not allowed.\"\r\n  }\r\n}"},{"code":"MethodNotAllowed","message":"{\r\n  \"error\": {\r\n    \"code\": \"OperationNotAllowed\",\r\n    \"message\": \"Operations on Secret Type ManagedCertificate are not allowed.\"\r\n  }\r\n}"},{"code":"MethodNotAllowed","message":"{\r\n  \"error\": {\r\n    \"code\": \"OperationNotAllowed\",\r\n    \"message\": \"Operations on Secret Type ManagedCertificate are not allowed.\"\r\n  }\r\n}"}]}}

Additional context

By minimum I need to find a way where most front door settings are managed through bicep script and deployment succeeds without
touching the custom domain to endpoint/route mappings. Script needs to work also when run from Azure DevOps pipeline.

I can see this might be rather an ARM issue and not bicep issue, but wanted also to get this documented with the latest bicep API's that one gets when converting from ARM templates currently. And on the other hand I would like to find a workaround to make the bicep not touch the custom domains if it is not possible to do this configuration fully yet.

My current bicep template that produces similar error is as follows (with some names changed). Also note that I am actually setting three different endpoints and hence have implemented some looping, but suspecting that is not relevant for for reproducing this issue. Providing this cleaned up version of the script mainly for discussing the topic more easily and to improve ability for others to find this issue if they are using same API versions (lots of examples and issues are using some older API versions that also have totally different namespaces and names).

@allowed(['development', 'production'])
param environmentType string = 'development'
param profile_name string = 'myproduct-fd-${environmentType}'
param endpoint_names array = ['myproduct-app-${environmentType}', 'myproduct-app-mg-${environmentType}', 'myproduct-app-pl-${environmentType}']
@description('Custom domains array that needs to contain same amount of custom domains as there are endpoint_names and they must be in correct order for those endpoins.')
param customDomains array = environmentType == 'production' ? ['api.mydomain.com', 'management.mydomain.com', 'app.mydomain.com'] : ['api-dev.mydomain.com', 'management-dev.mydomain.com', 'app-dev.mydomain.com']
param tags object = {
  'project': 'myproduct'
  'usage': environmentType
  'Environment': 'myproduct-env-${environmentType}'
}

@allowed(['weu', 'usn'])
param regionShorthand string = 'weu'

// Hard coded guid that was setup when manually deploying dev env.
var secretGuid = 'a4611ac0-6ab8-88fb-82ab-426556cfa890'

resource profiles_myproduct_fd_resource 'Microsoft.Cdn/profiles@2021-06-01' = {
  name: profile_name
  location: 'Global'
  tags: tags
  sku: {
    name: 'Standard_AzureFrontDoor'
  }
  properties: {
    originResponseTimeoutSeconds: 60
  }
}

resource certificate_resources 'Microsoft.Cdn/profiles/secrets@2021-06-01' = [for domainName in customDomains: {
  parent: profiles_myproduct_fd_resource
  name: '${secretGuid}-${replace(domainName, '.', '-')}'
  properties: {
    parameters: {
      type: 'ManagedCertificate'
    }
  }
}]


resource custom_domain_resources 'Microsoft.Cdn/profiles/customdomains@2021-06-01' = [for (domainName, i) in customDomains: {
  parent: profiles_myproduct_fd_resource
  name: replace(domainName, '.', '-')
  properties: {
    hostName: domainName
    tlsSettings: {
      certificateType: 'ManagedCertificate'
      minimumTlsVersion: 'TLS12'
      secret: {
        id: certificate_resources[i].id
      }
    }
  }
}]

resource fd_endpoint_resources 'Microsoft.Cdn/profiles/afdendpoints@2021-06-01' = [for endpoint_name in endpoint_names: {
  parent: profiles_myproduct_fd_resource
  name: endpoint_name
  location: 'Global'
  properties: {
    enabledState: 'Enabled'
  }
}]

resource fd_origin_groups 'Microsoft.Cdn/profiles/origingroups@2021-06-01' = [for (endpoint_name, i) in endpoint_names: {
  parent: profiles_myproduct_fd_resource
  name: '${endpoint_name}-origin-group'
  properties: {
    loadBalancingSettings: {
      sampleSize: 4
      successfulSamplesRequired: 3
      additionalLatencyInMilliseconds: 50
    }
    healthProbeSettings: {
      probePath: '/api/healthz'
      probeRequestType: 'GET'
      probeProtocol: 'Https'
      probeIntervalInSeconds: 100
    }
    sessionAffinityState: 'Disabled'
  }
}]

var conditionalShorthand = regionShorthand == 'weu' ? '' : '-${regionShorthand}'

resource fd_origins 'Microsoft.Cdn/profiles/origingroups/origins@2021-06-01' = [for (endpoint_name, i) in endpoint_names: {
  parent: fd_origin_groups[i]
  name: '${endpoint_name}-${regionShorthand}-origin'
  properties: {
    hostName: '${endpoint_name}${conditionalShorthand}.azurewebsites.net'
    httpPort: 80
    httpsPort: 443
    originHostHeader: '${endpoint_name}${conditionalShorthand}.azurewebsites.net'
    priority: 1
    weight: 1000
    enabledState: 'Enabled'
    enforceCertificateNameCheck: true
  }
}]

resource fd_routes 'Microsoft.Cdn/profiles/afdendpoints/routes@2021-06-01' = [for (endpoint_name, i) in endpoint_names: {
  parent: fd_endpoint_resources[i]
  name: 'default-route'
  properties: {
    customDomains: [
      {
        id: custom_domain_resources[i].id
      }
    ]
    originGroup: {
      id: fd_origin_groups[i].id
    }
    ruleSets: []
    supportedProtocols: [
      'Https'
    ]
    patternsToMatch: [
      '/*'
    ]
    forwardingProtocol: 'MatchRequest'
    linkToDefaultDomain: 'Enabled'
    httpsRedirect: 'Enabled'
    enabledState: 'Enabled'
  }
}]

Before adding the customDomains to routes I tried omitting the customDomains parameter altogether from the routes to maybe get not modify that custom domain to route mapping at all, but the mapping still got cleared.

Now that the custom domains were added to the script, a fresh deployment with the script fails to get the routes created while still completing e.g. the endpoint setup. This I still need to find a workaround too, to get FD deployed to production.

[alex-frankel]

FWIW, this does not seem like a bicep specific issue, but it would be good to take a deeper look. @brwilkinson do you have some time to look into this?

@aviita -- it may be faster to open an azure support ticket if that is an option for you.

[brwilkinson]

@aviita

Please take a look here for some examples.

sample input

https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/tenants/AOA/ACU1.T5.parameters.json#L793

sample templates

https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/FD.CDN.bicep
https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/FD.CDN-Profiles.bicep
https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/FD.CDN-Profiles-AFDEP.bicep
https://github.com/brwilkinson/AzureDeploymentFramework/blob/main/ADF/bicep/FD.CDN-Profiles-AFDEP-EP.bicep

These will deploy The profile + afdendpoints and create the customDNS and Routes.

Depending on your capability to generate the TXT verify records during your deployment, you may have the following in the Domains section.

You can then capture the required fields and manually or via an external process setup the TXT record as well as the CName record.

[brwilkinson]

If you take a look at the customDomains definition, you can see that secret should be read only.

You also do not need to define the secret resource this is managed by the platform.

resource customDomains 'Microsoft.Cdn/profiles/customDomains@2021-06-01' = {
  name: toLower(replace('${ep.name}.${ep.zone}', '.', '-'))
  parent: CDNProfile
  properties: {
    hostName: toLower('${ep.name}.${ep.zone}')
    tlsSettings: {
      certificateType: 'ManagedCertificate'
      minimumTlsVersion: 'TLS12'
      // secret: {
      //   id: 
      // }
    }
  }
  dependsOn: [
    DNSCNAME
  ]
}

If you remove both of those then you should be good.

[aviita]

Thanks for quick replies @alex-frankel and @brwilkinson. I think I am going to need to leave this on hold for my summer vacation.

I am leaving for summer vacation and do not want to rock the boat just before that. If this messes up the existing domains somehow again, I would be risking having to consult support for our DNS provider.

This way of verification in palace for front door seems to mess up the DNS providers system. These entries get some subfolders created in their system and none of the related DNS records are visible to us. So I assume no need for bicep deployments on the front door during July and will pick up on our internal bug regarding this in August.

Anyways we have the front door already setup for production too and bicep was great help there regardless of this problem. Much more enjoyable to work with than ARM templates.

[aviita]

The closing of this issue was anyways accidental. Not sure if you can keep this parked in your system for a month.

Anyways I could still mention that the what-if now shows it would delete properties.tlsSettings.secret and the related id:
id: "/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/myproduct-rg-shared-development/providers/Microsoft.Cdn/Profiles/myproduct-fd-development/secrets/-api-dev-mydomain-com"

So that is why I want to play it safe.

[brwilkinson]

The whatif issue is also tracked in

• Resource Type to get existing certificates from Microsoft.KeyVault is missing #7354

I will update he notes there to include this specific example.

Feel free to close this, if you have a working solution, other than the whatif issue.

[brwilkinson]

will close here, however please follow on #7354

[aviita]

I dared to run the az deployment create with updated bicep script now and yes, it does seem to work now. Thanks a lot for your quick and accurate response.

After the deployment create, what-if still shows that delete for properties.tlsSettings.secret in Microsoft.Cdn/profiles customDomains, but this is fine for me for now as long as we know it and it does not actually affect the functionality.