Bicep-generated nested templates causing many 'Cannot bind argument to parameter TemplateText' errors in Marketplace tests

[NickSpag]

Background

• We're using the latest arm-ttk alongside our Azure application offer per Partner Center requirements.
• We generate our mainTemplate.json from Azure Bicep files, which create nested templates

Issues

Broadly, we see a littany of issues that we think are related to nested templates. They most commonly surface by erroring certain tests with a Cannot bind argument to parameter 'TemplateText' because it is (an empty <type>|null) but there are also unreferenced variable or empty property warnings.

But we also see multiple outright errors during ingestion that usually point to lines 428, 711, 775:

InvalidOperation: /Users/nickspagnola/Development/Projects/arm-ttk/arm-ttk/Test-AzTemplate.ps1:428
Line |
428 |  …             $location.Line += $testOut.InnerTemplateLocation.Line - 1 …|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| The property 'Line' cannot be found on this object. Verify that the property exists and can be set.

Expand-AzTemplate: /Users/nickspagnola/Development/Projects/arm-ttk/arm-ttk/Test-AzTemplate.ps1:711
Line |
711 |  …      $expandedTemplate =Expand-AzTemplate -TemplatePath $templatePath|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Could not extract inner templates for './working-app'.

Write-Error: /Users/nickspagnola/Development/Projects/arm-ttk/arm-ttk/Test-AzTemplate.ps1:775
Line |
775 |                  Test-FileList # we just call it directly.|
~~~~~~~~~~~~~
| Could not extract inner templates for './working-app'.

We amended the arm-ttk scripts to print out which template was responsible for the error, and its typically in areas where our bicep modules are nested two or three times deep.

Impact

While we think this is a broader issue related to nested templates, it just so happens that it is hitting a required Marketplace test- Location Should Not Be Hardcoded. But we do not hardcode any locations. I've changed any references to locations (like in urls in deployment scripts, azure app config key values, etc) to use only the top level parameter provided by the createUiDefinition.json. There are no words in our mainTemplate.json that use an azure region. But the error persists.

Thank you!

[NickSpag]

In an effort to isolate the issue I've been able to reproduce the InvalidOperation error in as small of a case as possible, and it's created some interesting behavior:

In this case, we're running arm-ttk with this command:

TESTAZTEMPLATE_ARGS="Test-AzMarketplacePackage -TemplatePath $TARGET_DIRECTORY/mainTemplate.json"
pwsh -noprofile -nologo -command "Import-Module '${LOCAL_ARMTTK_ROOT}/arm-ttk/arm-ttk.psd1'; ${TESTAZTEMPLATE_ARGS} ; if (\$error.Count) { exit 1}"

--

There are three bicep files that generate a single mainTemplate.json, with the following structure and resources:

• main.bicep (a call to keyvaults.bicep, and an appconfig resource that, if removed, removes the error)
  • keyvaults.bicep (a call to keyvault.bicep, and a commented out keyvault resource that, if added, removes the error)
    • keyvault.bicep (a keyvault resource)

Files

main.bicep

param location string

// if you remove this, the error will not happen
resource azureAppConfig 'Microsoft.AppConfiguration/configurationStores@2022-05-01' = {
  name: '$appconfig'
  location: location
  sku: {
    name: 'standard'
  }
  properties: {
    publicNetworkAccess: 'Enabled'
  }
}

// nested module
module keyVaults 'keyvaults.bicep' = {
  name: 'keyvaults-module'
  params: {
    keyVaultName: 'keyvaultname'
  }
}

keyvaults.bicep

param keyVaultName string

module bots 'keyvault.bicep' = {
  name: 'keyvault-module'
  params:{
    keyVaultName: keyVaultName
  }
}

// curiously, if you add this the error goes away
// resource keyVaultName_appPasswordSecret 'Microsoft.KeyVault/vaults/secrets@2019-09-01' = {
//   name: '${keyVaultName}/appPasswordSecret1'
//   properties: {
//     attributes: {
//       enabled: true
//     }
//     value: 'blah'
//   }
// }

keyvault.bicep

param keyVaultName string

// including this triggers the error
resource keyVaultName_appPasswordSecret 'Microsoft.KeyVault/vaults/secrets@2019-09-01' = {
  name: '${keyVaultName}/appPasswordSecret'
  properties: {
    attributes: {
      enabled: true
    }
    value: 'blah'
  }
}

Produces this mainTemplate.json:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "metadata": {
    "_generator": {
      "name": "bicep",
      "version": "0.10.61.36676",
      "templateHash": "17942593897169903016"
    }
  },
  "parameters": {
    "location": {
      "type": "string"
    }
  },
  "resources": [
    {
      "type": "Microsoft.AppConfiguration/configurationStores",
      "apiVersion": "2022-05-01",
      "name": "$appconfig",
      "location": "[parameters('location')]",
      "sku": {
        "name": "standard"
      },
      "properties": {
        "publicNetworkAccess": "Enabled"
      }
    },
    {
      "type": "Microsoft.Resources/deployments",
      "apiVersion": "2020-10-01",
      "name": "keyvaults-module",
      "properties": {
        "expressionEvaluationOptions": {
          "scope": "inner"
        },
        "mode": "Incremental",
        "parameters": {
          "keyVaultName": {
            "value": "keyvaultname"
          }
        },
        "template": {
          "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
          "contentVersion": "1.0.0.0",
          "metadata": {
            "_generator": {
              "name": "bicep",
              "version": "0.10.61.36676",
              "templateHash": "5774184200920166479"
            }
          },
          "parameters": {
            "keyVaultName": {
              "type": "string"
            }
          },
          "resources": [
            {
              "type": "Microsoft.Resources/deployments",
              "apiVersion": "2020-10-01",
              "name": "keyvault-module",
              "properties": {
                "expressionEvaluationOptions": {
                  "scope": "inner"
                },
                "mode": "Incremental",
                "parameters": {
                  "keyVaultName": {
                    "value": "[parameters('keyVaultName')]"
                  }
                },
                "template": {
                  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
                  "contentVersion": "1.0.0.0",
                  "metadata": {
                    "_generator": {
                      "name": "bicep",
                      "version": "0.10.61.36676",
                      "templateHash": "17207296097468162821"
                    }
                  },
                  "parameters": {
                    "keyVaultName": {
                      "type": "string"
                    }
                  },
                  "resources": [
                    {
                      "type": "Microsoft.KeyVault/vaults/secrets",
                      "apiVersion": "2019-09-01",
                      "name": "[format('{0}/appPasswordSecret', parameters('keyVaultName'))]",
                      "properties": {
                        "attributes": {
                          "enabled": true
                        },
                        "value": "blah"
                      }
                    }
                  ]
                }
              }
            }
          ]
        }
      }
    }
  ]
}

Files here: arm-ttk-repro.zip

Error

It will create one error:

InvalidOperation: /Users/nickspagnola/Development/Projects/arm-ttk/arm-ttk/Test-AzTemplate.ps1:428
Line |
428 |  …             $location.Line += $testOut.InnerTemplateLocation.Line - 1 …|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| The property 'Line' cannot be found on this object. Verify that the property exists and can be set.

Notes

• This is just a subset of the errors we see in our broader solution.
• I haven't tried many other resources besides a keyvault, but I have tried others that are similar (azure app config values that also construct their parent resource with a '{parentResourceName}/resourceType' syntax) and it does not produce the error.
• I've isolated the mainTemplate here and ran arm-ttk against it directly, but if I add a basic createUiDefinition.json and run tests against the directory, the Parameters Without Default Must Exist In CreateUIDefinition test will fail twice on the keyVaultName parameter in keyvaults.bicep and keyvault.bicep. Not sure if that is intended behavior, but it forces us to add defaults to all of our nested tempaltes in bicep which eliminates some of the type safety benefits and indicates there may be confusion in the nested template graph. edit: I see Nested templates and Parameters Without Default Must Exist In CreateUIDefinition #681 is pointing out this issue as well
• At the end of main.bicep, if I replaced the keyvaultname string with an empty string for the keyvaults.bicep argument, it creates a Cannot bind argument to parameter 'Match' because it is an empty string. error in the Parameter Types Should Be Consistent test. Again not sure if that's intended behavior, since in this case it would be catching something problematic, but not exactly reporting the test failure properly.

If there is anything else that we can provide or any other way to help, we're happy to hear it. If you also suspect a certain area of the arm-ttk source we'd be happy to get pointed in the right direction. thanks!

[StartAutomating]

@bmoore-msft this looks like it may impact inner templates, and I'd like to take a look.

[bmoore-msft]

may be related to #686

[NickSpag]

I tested with the latest master and can confirm the isolated case I highlighted above is resolved with the #690 fixes.

The other two errors remain in our broader templates, and I'll look to try to isolate them as well:

Expand-AzTemplate: /Users/nickspagnola/Development/Projects/arm-ttk/arm-ttk/Test-AzTemplate.ps1:721
Line |
721 |  …      $expandedTemplate =Expand-AzTemplate -TemplatePath $templatePath|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Could not extract inner templates for './working-app'.

Write-Error: /Users/nickspagnola/Development/Projects/arm-ttk/arm-ttk/Test-AzTemplate.ps1:785
Line |
785 |                  Test-FileList # we just call it directly.|
~~~~~~~~~~~~~
| Could not extract inner templates for './working-app'.

If these errors point to any type of pattern I should be looking at or anyone thinks they have an idea what might trigger it to help produce a repro case, would be happy to hear it. thank you

[bmoore-msft]

@NickSpag - are you still seeing the errors with the 0.16 release on mainTemplate.json above? I ran local and I don't see the errors with 0.16...

[NickSpag]

@bmoore-msft nope, the latest fixes in 0.16 look good: the The property 'Line' cannot be found on this object... error is no longer happening on both the isolated repro I pasted above and with our larger real world templates.

However in those real world templates we are still seeing the two other different types of errors that I mentioned, at time of ingestion, that result in the TemplateText error in the Location Should Not Be Hardcoded test. I'm going to try to find time to go through again and try to isolate another pattern that reliably creates either or both of those errors.

[bmoore-msft]

Ah, I think I'm with you... so it's "fixed" when you run locally, but during Marketplace publishing you're still seeing the errors? If so, it's likely that the marketplace is still using the older version of the TTK...

[NickSpag]

@bmoore-msft no, I mean to say that only one type of error is fixed.

When I first posted this, running arm-ttk on our real mainTemplate.json and createUiDefinition.json we use for our Marketplace managed app created three different types of errors. I went through our template and just commented out as much as possible while preserving the errors. From that, I was able to create the very narrow example biceps and generated mainTemplate.json in this post to isolate one of those three error types. When running the latest arm-ttk on that example mainTemplate.json locally, that specific error that it reproduced is now gone. It's also gone when running it against our larger real mainTemplate. However, the other two types of errors I highlighted here still persist. My guess is I'll have to go through our template again and try to narrow down what combination of resources/pattern is creating those other two error types so I can create another example that will reproduce them.

[bmoore-msft]

It sounds like you may have as many as 3 bugs here only one of which has been fixed in the latest release. If you don't mind, I'm going to close this one out and if you can narrow down the repro can you open 1 (or 2) new bugs as appropriate?

[NickSpag]

Will do

[bkanda]

@NickSpag by any chance have you got the below issue fixed.

• Could not extract inner templates &
• Cannot bind argument to parameter 'TemplateText' because it is an empty string

[NickSpag]

@bkanda As of version 0.23 we still see multiple occurrences of the same two errors related to inner templates, when you first run arm-ttk (before any specific tests). we think something about the way bicep creates nested arm templates makes them difficult for arm-ttk to ingest:

Expand-AzTemplate: /Users/nickspagnola/Development/Zammo/.nuke/temp/ArmTtk/arm-ttk/Test-AzTemplate.ps1:721
Line |
 721 |  …      $expandedTemplate =Expand-AzTemplate -TemplatePath $templatePath
     | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Could not extract inner templates for
     | '/Users/nickspagnola/Development/Zammo/output/iac-app-manifest/development'.

Write-Error: /Users/nickspagnola/Development/Zammo/.nuke/temp/ArmTtk/arm-ttk/Test-AzTemplate.ps1:785
Line |
 785 |                  Test-FileList # we just call it directly.
     | ~~~~~~~~~~~~~
     | Could not extract inner templates for
     | '/Users/nickspagnola/Development/Zammo/output/iac-app-manifest/development'.

Cannot bind argument to parameter 'TemplateText' usually just surfaces in a specific test, most likely an error when trying to reference a template that wasn't ingested because of the previous errors. It's not consistent in how it appears.

Unfortunately I haven't had time to try to create another minimum reproduction. Since the Azure Marketplace/Partner Center (the reason we use arm-ttk) is not guaranteed to use the latest version, and arm-ttk updates being much more sparse, it's probably not going to be worth investigating for us, sorry. Would love to see anything you were able to reproduce though- gotta be some specific pattern that's creating the issue.