Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

-j/--handle-extended-json-format does not work as expected for mutil-line string containing double quotes #12909

Closed
galiacheng opened this issue Apr 7, 2020 · 16 comments
Assignees
Labels
ARM - Templates ARM az resource/group/lock/tag/deployment/policy/managementapp/account management-group Service Attention This issue is responsible by Azure service team. Template Parse The issues of template parse
Milestone

Comments

@galiacheng
Copy link

Hi experts,
I found if an ARM template contains multi-line string containing double quotes, the template deployment will fail, please help to check, thank you.
I would like to pass parameters that contains double quotes to shell script within virtual machine extension, please see variables executeCustomScriptArguments1. The template deployment failed. I deploy the template with az cli flag --handle-extended-json-format, az version is 2.2.0.
My template works if I change the mutil-line string to single line string.
Error:

cli.azure.cli.core.util : Invalid control character at: line 1 column 5789 (char 5788)

The flag also works for cases listed in document, I can make sure my Az CLI supports --handle-extended-json-format.

Attach my template:

{
   "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
   "contentVersion": "1.0.0.0",
   "parameters": {
      "_artifactsLocation": {
         "type": "string",
         "metadata": {
            "description": "The base URI where artifacts required by this template are located. When the template is deployed using the accompanying scripts, a private location in the subscription will be used and this value will be automatically generated."
         },
         "defaultValue": "[deployment().properties.templateLink.uri]"
      },
      "_artifactsLocationSasToken": {
         "type": "securestring",
         "metadata": {
            "description": "The sasToken required to access _artifactsLocation.  When the template is deployed using the accompanying scripts, a sasToken will be automatically generated. Use the defaultValue if the staging location is not secured."
         },
         "defaultValue": ""
      },
      "guidValue": {
         "type": "string",
         "defaultValue": "[newGuid()]"
      },
      "adminUsername": {
         "type": "string",
         "metadata": {
            "description": "User name for the Virtual Machine."
         }
      },
      "authenticationType": {
         "type": "string",
         "defaultValue": "password",
         "allowedValues": [
            "sshPublicKey",
            "password"
         ],
         "metadata": {
            "description": "Type of authentication to use on the Virtual Machine. SSH key is recommended."
         }
      },
      "adminPasswordOrKey": {
         "type": "securestring",
         "metadata": {
            "description": "SSH Key or password for the Virtual Machine. SSH key is recommended."
         }
      },
      "acceptOTNLicenseAgreement": {
         "type": "string",
         "metadata": {
            "description": "Do you agree to provide OTN credentials to accept OTN License Agreement? Enter Y or y to agree, else N or n"
         }
      },
      "otnAccountUsername": {
         "type": "string",
         "metadata": {
            "description": "Username for your Oracle Technology Network account"
         }
      },
      "otnAccountPassword": {
         "type": "securestring",
         "metadata": {
            "description": "Password for your Oracle Technology Network account"
         }
      },
      "portsToExpose": {
         "type": "string",
         "defaultValue": "80,443,7001-9000",
         "metadata": {
            "description": "Ports and port ranges to expose"
         }
      },
      "wlsDomainName": {
         "type": "string",
         "metadata": {
            "description": "Provide Weblogic domain name"
         }
      },
      "wlsUserName": {
         "type": "string",
         "metadata": {
            "description": "Username for your Weblogic domain name"
         }
      },
      "wlsPassword": {
         "type": "securestring",
         "metadata": {
            "description": "Password for your Weblogic domain name"
         }
      },
      "dnsLabelPrefix": {
         "type": "string",
         "metadata": {
            "description": "Unique DNS Name for the Public IP used to access the Virtual Machine."
         }
      },
      "linuxOSVersion": {
         "type": "string",
         "defaultValue": "7.4",
         "allowedValues": [
            "7.4",
            "7.3"
         ],
         "metadata": {
            "description": "The Oracle Linux version for the VM. This will pick a fully patched image of this given Oracle Linux version."
         }
      },
      "location": {
         "type": "string",
         "defaultValue": "[resourceGroup().location]",
         "metadata": {
            "description": "Location for all resources."
         }
      },
      "vmSizeSelect": {
         "type": "string",
         "defaultValue": "Standard_A3",
         "allowedValues": [
            "Standard_A1",
            "Standard_A2",
            "Standard_A3",
            "Standard_A4"
         ],
         "metadata": {
            "description": "Select appropriate VM Size as per requirement (Standard_A1, Standard_A2, Standard_A3, Standard_A4)"
         }
      },
      "configureAAD": {
         "type": "bool",
         "defaultValue": false,
         "metadata": {
            "description": "Bool value, if it's set to true, will enable Azure Active Directory after WebLogic Server starts."
         }
      },
      "wlsLDAPProviderName": {
         "type": "string",
         "defaultValue": "AzureActiveDirectoryProvider",
          "metadata": {
            "description": "The value used for creating authentication provider name of WebLogic Server."
         }
      },
      "addsServerHost": {
         "type": "string",
         "defaultValue": "ldaps.example.com",
         "metadata": {
            "description": "The LDAP server host."
         }
      },
      "addsPublicIP": {
         "type": "string",
         "defaultValue": "The LDAP server public IP address"
      },
      "aadsPortNumber": {
         "type": "string",
         "defaultValue": "636",
         "metadata": {
            "description": "Accessible port of the LDAP server."
         }
      },
      "wlsLDAPPrincipal": {
         "type": "string",
         "defaultValue": "null",
         "metadata": {
            "description": "The Distinguished Name (DN) of the LDAP user that WebLogic Server should use to connect to the LDAP server."
         }
      },
      "wlsLDAPPrincipalPassword": {
         "type": "securestring",
         "defaultValue":"[newGuid()]",
         "metadata": {
            "description": "The credential (usually a password) used to connect to the LDAP server."
         }
      },
      "wlsLDAPUserBaseDN": {
         "type": "string",
         "defaultValue": "null",
         "metadata": {
            "description": "The base distinguished name (DN) of the tree in the LDAP directory that contains users."
         }
      },
      "wlsLDAPGroupBaseDN": {
         "type": "string",
         "defaultValue": "null",
         "metadata": {
            "description": "The base distinguished name (DN) of the tree in the LDAP directory that contains groups."
         }
      },
      "wlsLDAPSSLCertificate": {
         "type": "string",
         "defaultValue": "null",
         "metadata": {
            "description": "Client certificate that will be imported to trust store of SSL."
         }
      }
   },
   "variables": {
      "storageAccountName": "[concat(take(replace(parameters('guidValue'),'-',''),6),'olvm')]",
      "imagePublisher": "Oracle",
      "imageOffer": "Oracle-Linux",
      "nicName": "adminServerVM_NIC",
      "addressPrefix": "10.0.0.0/16",
      "subnetName": "Subnet",
      "subnetPrefix": "10.0.0.0/24",
      "storageAccountType": "Standard_LRS",
      "publicIPAddressName": "adminServerVM_PublicIP",
      "publicIPAddressType": "Dynamic",
      "vmName": "adminServerVM",
      "vmSize": "[parameters('vmSizeSelect')]",
      "virtualNetworkName": "[concat(parameters('wlsDomainName'),'_VNET')]",
      "oradownScript": "oradown.sh",
      "ScriptFileName": "setupAdminDomain.sh",
      "pyConfigureSSL": "configure-ssl.py",
      "pyConfigureLDAP": "configure-active-directory.py",
      "linuxConfiguration": {
         "disablePasswordAuthentication": true,
         "ssh": {
            "publicKeys": [
               {
                  "path": "[concat('/home/', parameters('adminUsername'), '/.ssh/authorized_keys')]",
                  "keyData": "[parameters('adminPasswordOrKey')]"
               }
            ]
         }
      },
      "subnetRef": "[resourceId('Microsoft.Network/virtualNetworks/subnets', variables('virtualNetworkName'), variables('subnetName'))]",
      "networkSecurityGroupName": "[concat(parameters('dnsLabelPrefix'), '-nsg')]",
      "networkSecurityGroupRef": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('networkSecurityGroupName'))]",
      "requiredPortrange": ",65200-65535,5556",
      "executeCustomScriptArguments1": "[
         if(parameters('configureAAD'), 
         concat(parameters('wlsLDAPProviderName'), ' ',  
         parameters('addsServerHost'), ' ', 
         parameters('aadsPortNumber'), ' ', 
         concat('\"',parameters('wlsLDAPPrincipal'),'\"'), ' ', 
         parameters('wlsLDAPPrincipalPassword'), ' ', 
         concat('\"',parameters('wlsLDAPUserBaseDN'),'\"'), ' ', 
         concat('\"',parameters('wlsLDAPGroupBaseDN'),'\"'), ' ', 
         parameters('wlsLDAPSSLCertificate'), ' ', 
         parameters('addsPublicIP'), ' ', 
         parameters('adminPasswordOrKey')),'')
         ]"
   },
   "resources": [
      {
         "type": "Microsoft.Network/networkSecurityGroups",
         "apiVersion": "2019-06-01",
         "name": "[variables('networkSecurityGroupName')]",
         "location": "[parameters('location')]",
         "properties": {
            "securityRules": [
               {
                  "name": "WebLogicPorts",
                  "properties": {
                     "protocol": "TCP",
                     "sourcePortRange": "*",
                     "sourceAddressPrefix": "*",
                     "destinationAddressPrefix": "*",
                     "access": "Allow",
                     "priority": 320,
                     "direction": "Inbound",
                     "destinationPortRanges": "[split(concat(parameters('portsToExpose'),variables('requiredPortrange')), ',')]"
                  }
               }
            ]
         }
      },
      {
         "type": "Microsoft.Storage/storageAccounts",
         "apiVersion": "2019-06-01",
         "name": "[variables('storageAccountName')]",
         "location": "[parameters('location')]",
         "sku": {
            "name": "[variables('storageAccountType')]"
         },
         "kind": "Storage",
         "properties": {}
      },
      {
         "type": "Microsoft.Network/publicIPAddresses",
         "apiVersion": "2018-11-01",
         "name": "[variables('publicIPAddressName')]",
         "location": "[parameters('location')]",
         "properties": {
            "publicIPAllocationMethod": "[variables('publicIPAddressType')]",
            "dnsSettings": {
               "domainNameLabel": "[concat(toLower(parameters('dnsLabelPrefix')),'-',take(replace(parameters('guidValue'), '-', ''), 10),'-',toLower(parameters('wlsDomainName')))]"
            }
         }
      },
      {
         "type": "Microsoft.Network/virtualNetworks",
         "apiVersion": "2018-11-01",
         "name": "[variables('virtualNetworkName')]",
         "location": "[parameters('location')]",
         "dependsOn": [
            "[variables('networkSecurityGroupRef')]"
         ],
         "properties": {
            "addressSpace": {
               "addressPrefixes": [
                  "[variables('addressPrefix')]"
               ]
            },
            "subnets": [
               {
                  "name": "[variables('subnetName')]",
                  "properties": {
                     "addressPrefix": "[variables('subnetPrefix')]",
                     "networkSecurityGroup": {
                        "id": "[variables('networkSecurityGroupRef')]"
                     }
                  }
               }
            ]
         }
      },
      {
         "type": "Microsoft.Network/networkInterfaces",
         "apiVersion": "2018-11-01",
         "name": "[variables('nicName')]",
         "location": "[parameters('location')]",
         "dependsOn": [
            "[resourceId('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]",
            "[resourceId('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]"
         ],
         "properties": {
            "ipConfigurations": [
               {
                  "name": "ipconfig1",
                  "properties": {
                     "privateIPAllocationMethod": "Dynamic",
                     "publicIPAddress": {
                        "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"
                     },
                     "subnet": {
                        "id": "[variables('subnetRef')]"
                     }
                  }
               }
            ]
         }
      },
      {
         "type": "Microsoft.Compute/virtualMachines",
         "apiVersion": "2018-10-01",
         "name": "[variables('vmName')]",
         "location": "[parameters('location')]",
         "dependsOn": [
            "[resourceId('Microsoft.Storage/storageAccounts/', variables('storageAccountName'))]",
            "[resourceId('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
         ],
         "properties": {
            "hardwareProfile": {
               "vmSize": "[variables('vmSize')]"
            },
            "osProfile": {
               "computerName": "[variables('vmName')]",
               "adminUsername": "[parameters('adminUsername')]",
               "adminPassword": "[parameters('adminPasswordOrKey')]",
               "linuxConfiguration": "[if(equals(parameters('authenticationType'), 'password'), json('null'), variables('linuxConfiguration'))]"
            },
            "storageProfile": {
               "imageReference": {
                  "publisher": "[variables('imagePublisher')]",
                  "offer": "[variables('imageOffer')]",
                  "sku": "[parameters('linuxOSVersion')]",
                  "version": "latest"
               },
               "osDisk": {
                  "createOption": "FromImage"
               },
               "dataDisks": [
                  {
                     "diskSizeGB": 900,
                     "lun": 0,
                     "createOption": "Empty"
                  }
               ]
            },
            "networkProfile": {
               "networkInterfaces": [
                  {
                     "id": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
                  }
               ]
            },
            "diagnosticsProfile": {
               "bootDiagnostics": {
                  "enabled": true,
                  "storageUri": "[reference(resourceId('Microsoft.Storage/storageAccounts/', variables('storageAccountName')), '2019-06-01').primaryEndpoints.blob]"
               }
            }
         }
      },
      {
         "type": "Microsoft.Compute/virtualMachines/extensions",
         "name": "[concat(variables('vmName'),'/newuserscript')]",
         "apiVersion": "2019-07-01",
         "location": "[parameters('location')]",
         "dependsOn": [
            "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'))]"
         ],
         "properties": {
            "publisher": "Microsoft.Azure.Extensions",
            "type": "CustomScript",
            "typeHandlerVersion": "2.0",
            "autoUpgradeMinorVersion": true,
            "settings": {
               "fileUris": [
                  "[uri(parameters('_artifactsLocation'), concat('scripts/', variables('ScriptFileName'), parameters('_artifactsLocationSasToken')))]",
                  "[uri(parameters('_artifactsLocation'), concat('scripts/', variables('oradownScript'), parameters('_artifactsLocationSasToken')))]",
                  "[uri(parameters('_artifactsLocation'), concat('scripts/', variables('pyConfigureSSL'), parameters('_artifactsLocationSasToken')))]",
                  "[uri(parameters('_artifactsLocation'), concat('scripts/', variables('pyConfigureLDAP'), parameters('_artifactsLocationSasToken')))]"
               ],
               "commandToExecute": "[concat('sh',' ',variables('ScriptFileName'),' ',parameters('acceptOTNLicenseAgreement'),' ',parameters('otnAccountUsername'),' ',parameters('otnAccountPassword'),' ',parameters('wlsDomainName'),' ',parameters('wlsUserName'),' ',parameters('wlsPassword'),' ',variables('vmName'),' ',parameters('configureAAD'),' ',variables('executeCustomScriptArguments1'),' ')]"
            }
         }
      }
   ]
}
@qianwens qianwens added the ARM az resource/group/lock/tag/deployment/policy/managementapp/account management-group label Apr 7, 2020
@triage-new-issues triage-new-issues bot removed the triage label Apr 7, 2020
@qianwens
Copy link
Member

qianwens commented Apr 7, 2020

@zhoxing-ms , please help take a look.

@yonzhan yonzhan added this to the S169 - For Build milestone Apr 8, 2020
@yonzhan
Copy link
Collaborator

yonzhan commented Apr 8, 2020

add to S169

@yonzhan
Copy link
Collaborator

yonzhan commented May 2, 2020

add to S171

@zhoxing-ms
Copy link
Contributor

@galiacheng Hi, mutil-line is not supported in the semantics of JSON, so I understand that this issue is a feature request.

cli.azure.cli.core.util : Invalid control character at: line 1 column 5789 (char 5788)

This error was obtained from the validationin of CLI. I tried to make CLI ignore the line feed in a string when validating JSON, but I still found some problems: the ARM template in CLI is designed to send to Service must be unmodified, and Service will add the line break as text to the multi-line string, such as:

  • In ARM template:
"meta": {
    "type": "string",
    "metadata": {
        "description": "meta of the network security group."
    },
    "defaultValue": "ARM
    template
    test
    newline"
}
  • After service processing:
"meta": {
    "type": "String",
    "value": "ARM\r\n      template\r\n      test\r\n      newline"
}

The string "executeCustomScriptArguments1" you pass in is actually a string of arguments that need to be parsed, and the extra line breaks in the middle will obviously destroy the parsing logic of these arguments.

@zhoxing-ms
Copy link
Contributor

Because the ARM template passed from CLI to Service can't be modified, so this problem can only be solved by Service, while on CLI side passes through the multi-line string when verifying.
But if Service is going to solve this problem, it also needs to consider the following two points:

  1. In some scenarios, if some customers want to keep the line break in the multi-line format, but this change will treat it as a single line string, there may be breaking change.
  2. If we define a custom newline character, we may not be able to distinguish whether the customers is passing it in as text or as a custom newline character, and whether it has an impact on the customer's existing ARM template.

So ask the Service team to see if this feature can be properly supported.

@zhoxing-ms zhoxing-ms added the Service Attention This issue is responsible by Azure service team. label Jun 1, 2020
@ghost
Copy link

ghost commented Jun 1, 2020

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @armleads-azure.

@zhoxing-ms
Copy link
Contributor

By the way, I find that the processing of mutil-line string in PowerShell is to add newline characters and then compress it into a single line before requesting a Service, Such as:

"WebTest": "[concat(\r\n                '<WebTest Name=\"Health check\" Id=\"', guid(concat('health check-', variables('partnerInsights'))), \r\n                '\" Enabled=\"True\" ......

I think modifying the ARM template of customers before requesting a Service is not in line with the design and may destroys the customers's input to some extent.

@ghost
Copy link

ghost commented Jun 25, 2020

Multi-line may not be supported by vanilla JSON, but the ARM template documentation says that it is valid. I find the lack of multi-line support particularly frustrating when dealing with stored procs.
It works fine when testing with Powershell locally, but failed when I tried to deploy with Azure Resource Group Deployment task in ADO, which I believe makes the Service call.

@ghost
Copy link

ghost commented Jun 25, 2020

@galiacheng Hi, mutil-line is not supported in the semantics of JSON, so I understand that this issue is a feature request.

cli.azure.cli.core.util : Invalid control character at: line 1 column 5789 (char 5788)

This error was obtained from the validationin of CLI. I tried to make CLI ignore the line feed in a string when validating JSON, but I still found some problems: the ARM template in CLI is designed to send to Service must be unmodified, and Service will add the line break as text to the multi-line string, such as:

  • In ARM template:
"meta": {
    "type": "string",
    "metadata": {
        "description": "meta of the network security group."
    },
    "defaultValue": "ARM
    template
    test
    newline"
}
  • After service processing:
"meta": {
    "type": "String",
    "value": "ARM\r\n      template\r\n      test\r\n      newline"
}

The string "executeCustomScriptArguments1" you pass in is actually a string of arguments that need to be parsed, and the extra line breaks in the middle will obviously destroy the parsing logic of these arguments.

I'd also make the argument that the value after the service processing is precisely what is expected. The fact that it won't work is specific to the field. It should be perfectly valid to have a variable whose value includes newlines. You just can't use that variable in a context where the new line wouldn't be allowed.

For example, both of these values should be equivalent, and neither of them should work in a situation like this.

"variables": {
  "foo": "1\n2\n3",
  "bar": "1
2
3"
}

@zhoxing-ms
Copy link
Contributor

@dschmack Thank you for your feedback. I think this requires our service team to confirm whether automatically adding line breaks to the muti-line value is in line with the expected behavior.
If they think the design works as expected and can ensure that the effect of the original parameters is not affected by these line breaks, then CLI will let the ARM template containing muti-line pass the verification.

@wiktorn
Copy link
Contributor

wiktorn commented Oct 17, 2020

Referring to the comment from @zhoxing-ms in #15502

  1. If it is multi-line of parameter values, service may break the resolution of those values.
    For example:
"defaultValue": "
    [resourceGroup().location]
"

It will cause an error in the service:

{"error":{"code":"InvalidResourceLocation","message":"The specified location '\r\n        [resourceGroup().location]\r\n      ' is invalid. A location must consist of characters, whitespace, digit, or following symbols '(,)'."}}

This is perfectly fine with me, as it gives clear error message what is wrong. And there is easy workaround for that:

"defaultValue": "[
    resourceGroup().location
]"

And as I understand, there are other cases, where Service will change the content of the template (striping the comments, changing the lines etc.).

But is it really stopping from fixing the bug? Maybe what could be done until Service is fixed is just to:

  • allow for comments and multiline strings
  • warn the user, that the comments will be stripped, and multiline strings converted to one-liners on Service side

So the users will be aware and can consciously decide, if that's OK with them. I think that although it not solves fully the issue, it still improves the situation, especially that official documentation states that multiline strings and comments are supported.

I understand that if I want to use multi-line strings in my deployments, I should use Powershell instead of azure-cli? If so, maybe azure-cli should detect such situation and refer user to the Powershell instead?

@zhoxing-ms
Copy link
Contributor

So the users will be aware and can consciously decide, if that's OK with them. I think that although it not solves fully the issue, it still improves the situation, especially that official documentation states that multiline strings and comments are supported.

@wiktorn Thank you for your advice. If the Service does not reply recently, we will consider this option.
However, I think it would be better for Service to solve this problem directly, then that the problem can be completely solved by merging that PR provided by you.

I understand that if I want to use multi-line strings in my deployments, I should use Powershell instead of azure-cli? If so, maybe azure-cli should detect such situation and refer user to the Powershell instead?

In fact, the handling of this problem of PowerShell is not in line with our design, because they modify the content of the ARM template and then send it to the Service. The original intention of our design is not to modify the ARM template on the client side.
The essence of this problem is not the problem of CLI, but the problem of service. However, PowerShell is compatible with service bug, but this compatibility does not conform to our original design. #12909 (comment)

@zhoxing-ms
Copy link
Contributor

@ifeoluwaokunoren Could you please help to triage this issue?

@ghost
Copy link

ghost commented Oct 19, 2020

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @armleads-azure.

@zhoxing-ms
Copy link
Contributor

@apclouds @alex-frankel Could you please take a look at this issue?

@zhoxing-ms
Copy link
Contributor

This issue was fixed by #15502

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ARM - Templates ARM az resource/group/lock/tag/deployment/policy/managementapp/account management-group Service Attention This issue is responsible by Azure service team. Template Parse The issues of template parse
Projects
None yet
Development

No branches or pull requests

9 participants