Chef extension consistently fails on Ubuntu VM

[aarsan]

For the past few days, I have been unable to successfully enable the Chef extension on my VM. I receive the following error:

private key does not match certificate - Check log file for details

There is a similar issue posted here:

Azure/azure-linux-extensions#28

[aarsan]

If I use the same name of a VM that was recently provisioned, I get this error. However, if I use a completely different name, it seems to work. There seems to be some remnants of the old machine lingering around in Azure that is causing this.

[aweinograd]

I think this is an Azure thing. We have this issue with resource groups and try not to reuse names frequently

[siddheshwar-more]

Hello @aarsan @aweinograd
-We are verifying this issue.

[aarsan]

Thanks. Actually, I can confirm that even with new server names, I still get this error. Something is going on and in the past 2 days, out of roughly 15 attempts, I've only been able to successfully enable the Chef extension once.

[aarsan]

I had the api version for the Chef extension set to 2015-06-15. I changed it to 2015-05-01-preview and it succeeded twice in a row. Since then, it failed 3 more times and no more successes.

[aarsan]

Ok, I believe I have isolated the problem. I recently changed my deployment to include a load balancer (this is when all my troubles began). When I deploy my VM behind a load balancer, the Chef extension fails 95% of the time with the error:

private key does not match certificate - Check log file for details

I have now deployed 12 times in a row without the load balancer and every single one succeeded. I'm not sure where to begin troubleshooting this. I am happy to send logs over or do whatever I need to do to help resolve this. I will do another 10 or so deploys behind the load balancer for more testing. Stay tuned.

[aweinograd]

@aarsan, Were you able to resolve this?

[NimishaS]

Sorry closed this by mistake.

[Aliasgar16]

@aarsan I tried to reproduce this chef extension installation issue on load-balanced VMs but on all the VMs the installation went successful. Can you share more details like what process you followed to create the load-balanced VMs and detailed extension installation failure logs and any other detail which would be helpful to proceed further with this issue ?

[NimishaS]

@aarsan, we are waiting for your response as we couldn't reproduce it.

[adamedx]

@aarsan, please re-open this if you have additional information to share as @NimishaS asked. Right now we can't reproduce the problem on our end, so we're blocked from figuring out how to address this.

[rubeon]

I can reproduce this with the following templates / parameters.

Error message is as follows:

{
  "status": "Failed",
  "error": {
    "code": "ResourceDeploymentFailure",
    "message": "The resource operation completed with terminal provisioning state 'Failed'.",
    "details": [
      {
        "code": "VMExtensionProvisioningError",
        "message": "VM has reported a failure when processing extension 'LinuxChefClient'. Error message: \"private key does not match certificate - Check log file for details\"."
      }
    ]
  }
}

single-vm-chef.json:

{
  "$schema": "$schema: http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", 
  "contentVersion": "1.0.0.0", 
  "variables": {
    "vnetID": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkName'))]", 
    "adminPassword": "[parameters('adminPassword')]", 
    "machineName": "[parameters('machineName')]", 
    "sshPublicKeys": "[parameters('sshPublicKeys')]", 
    "imageSKU": "[parameters('imageSKU')]", 
    "nicName": "testNIC", 
    "publicIPAddressType": "Dynamic", 
    "vmSize": "[parameters('vmSize')]", 
    "subnetName": "subnet-management", 
    "newStorageAccountName": "[parameters('newStorageAccountName')]", 
    "subnetPrefix": "10.0.0.0/24", 
    "imagePublisher": "[parameters('imagePublisher')]", 
    "vmStorageAccountContainerName": "testvms", 
    "addressPrefix": "10.0.0.0/16", 
    "storageAccountType": "Standard_LRS", 
    "apiVersion": "2015-06-15", 
    "publicIPAddressName": "[parameters('publicIPAddressName')]", 
    "location": "[resourceGroup().location]", 
    "adminUserName": "[parameters('adminUserName')]", 
    "virtualNetworkName": "test-vnet", 
    "subnetRef": "[concat(variables('vnetID'), '/subnets/', variables('subnetName'))]", 
    "vmExtensionName": "LinuxChefClient", 
    "imageOffer": "[parameters('imageOffer')]", 
    "OSDiskName": "osdisk-ubuntu"
  }, 
  "parameters": {
    "adminPassword": {
      "type": "string"
    }, 
    "machineName": {
      "type": "string"
    }, 
    "chef_environment": {
      "type": "string"
    }, 
    "sshPublicKeys": {
      "type": "array"
    }, 
    "imageSKU": {
      "type": "string"
    }, 
    "chef_server_url": {
      "type": "string"
    }, 
    "newStorageAccountName": {
      "type": "string", 
      "metadata": {
        "description": "Admin User Name"
      }
    }, 
    "imagePublisher": {
      "type": "string"
    }, 
    "vmSize": {
      "type": "string"
    }, 
    "runlist": {
      "type": "string"
    }, 
    "imageOffer": {
      "type": "string"
    }, 
    "validation_key": {
      "type": "string"
    }, 
    "validation_client_name": {
      "type": "string"
    }, 
    "location": {
      "type": "string"
    }, 
    "adminUserName": {
      "type": "string"
    }, 
    "publicIPAddressName": {
      "type": "string"
    }, 
    "chef_node_name": {
      "type": "string"
    }
  }, 
  "resources": [
    {
      "location": "[parameters('location')]", 
      "type": "Microsoft.Storage/storageAccounts", 
      "properties": {
        "accountType": "[variables('storageAccountType')]"
      }, 
      "apiVersion": "[variables('apiVersion')]", 
      "name": "[parameters('newStorageAccountName')]"
    }, 
    {
      "location": "[variables('location')]", 
      "type": "Microsoft.Network/publicIPAddresses", 
      "properties": {
        "publicIPAllocationMethod": "[variables('publicIPAddressType')]", 
        "dnsSettings": {
          "domainNameLabel": "[toLower(concat(resourceGroup().name, '-','public-ip'))]"
        }
      }, 
      "apiVersion": "[variables('apiVersion')]", 
      "name": "[variables('publicIPAddressName')]"
    }, 
    {
      "location": "[variables('location')]", 
      "type": "Microsoft.Network/virtualNetworks", 
      "properties": {
        "subnets": [
          {
            "name": "[variables('subnetName')]", 
            "properties": {
              "addressPrefix": "[variables('subnetPrefix')]"
            }
          }
        ], 
        "addressSpace": {
          "addressPrefixes": [
            "[variables('addressPrefix')]"
          ]
        }
      }, 
      "apiVersion": "[variables('apiVersion')]", 
      "name": "[variables('virtualNetworkName')]"
    }, 
    {
      "name": "[variables('nicName')]", 
      "dependsOn": [
        "[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]", 
        "[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]"
      ], 
      "type": "Microsoft.Network/networkInterfaces", 
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipconfig1", 
            "properties": {
              "subnet": {
                "id": "[variables('subnetRef')]"
              }, 
              "privateIPAllocationMethod": "Dynamic", 
              "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"
              }
            }
          }
        ]
      }, 
      "apiVersion": "[variables('apiVersion')]", 
      "location": "[variables('location')]"
    }, 
    {
      "name": "[variables('machineName')]", 
      "dependsOn": [
        "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]", 
        "[concat('Microsoft.Storage/storageAccounts/', variables('newStorageAccountName'))]"
      ], 
      "type": "Microsoft.Compute/virtualMachines", 
      "properties": {
        "hardwareProfile": {
          "vmSize": "[variables('vmSize')]"
        }, 
        "storageProfile": {
          "imageReference": {
            "sku": "[variables('imageSKU')]", 
            "publisher": "[variables('imagePublisher')]", 
            "version": "latest", 
            "offer": "[variables('imageOffer')]"
          }, 
          "osDisk": {
            "caching": "ReadWrite", 
            "vhd": {
              "uri": "[concat('http://',parameters('newStorageAccountName'), '.blob.core.windows.net/',variables('vmStorageAccountContainerName'), '/', variables('OSDiskName'),'.vhd')]"
            }, 
            "createOption": "FromImage", 
            "name": "osdisk"
          }
        }, 
        "osProfile": {
          "adminUsername": "[variables('adminUsername')]", 
          "computerName": "[variables('machineName')]", 
          "linuxConfiguration": {
            "ssh": {
              "publicKeys": "[parameters('sshPublicKeys')]"
            }
          }, 
          "adminPassword": "[variables('adminPassword')]"
        }, 
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces', variables('nicName'))]"
            }
          ]
        }
      }, 
      "apiVersion": "[variables('apiVersion')]", 
      "location": "[variables('location')]"
    }, 
    {
      "name": "[concat(variables('vmName'), '/', variables('vmExtensionName'))]", 
      "dependsOn": [
        "[concat('Microsoft.Compute/virtualMachines/',variables('vmName'))]"
      ], 
      "type": "Microsoft.Compute/virtualMachines/extensions", 
      "properties": {
        "protectedSettings": {
          "validation_key": "[parameters('validation_key')]"
        }, 
        "publisher": "Chef.Bootstrap.WindowsAzure", 
        "typeHandlerVersion": "1207.12", 
        "type": "LinuxChefClient", 
        "settings": {
          "bootstrap_options": {
            "environment": "[parameters('chef_environment')]", 
            "validation_client_name": "[parameters('validation_client_name')]", 
            "chef_node_name": "[parameters('chef_node_name')]", 
            "chef_server_url": "[parameters('chef_server_url')]"
          }, 
          "runlist": "[parameters('runlist')]", 
          "client_rb": "ssl_verify_mode :verify_none\n"
        }
      }, 
      "apiVersion": "[variables('apiVersion')]", 
      "location": "[variables('location')]"
    }
  ]
}

single-vm-chef.parameters.json:

{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "adminPassword": {
      "value": "MyPassWord1"
    },
    "machineName": {
      "value": "test-vm-01"
    },
    "chef_environment": {
      "value": "test-deploys"
    },
    "sshPublicKeys": {
      "value": [
        {
          "path": "/home/demouser/.ssh/authorized_keys",
          "keyData": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAp8W4vSF1wr2JvU6rSr9nVclgcNCHYx+TS9CFa9qjFshPruLcGFbfQiT102Q/WeBhGNUHLQGmToNkOGyqJnJdjdQoNB7XeyYWIyxXBbPa9Zjj4+lgGUXhHsEiYhJtfqMZQvUuoKBgRFGmohJE+kpm9e6+pzpeWepBhnAIWq8sIQIoeoEIa8yl4UCmgxggTqZ9POLl1S4J5x9AN4cZgzZGtWNEiZnw+Ku04ASGqdXA9B869ilxCbhk7QmsZ6IUNr2bBd7Pd03e8zJgPRu3731feJmNgagNl42zxNEkp7L2pDquT2B0R5hKI8CYE1ogVq/tOFFHRms148FrqWOV93ufcw=="
        },
        {
          "path": "/home/demouser/.ssh/authorized_keys",
          "keyData": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuIi+Gr3ZrxQhyHx0bB1IpqZQ8Z0MgiCS1rVcilWA3LI01iPD5XIxwktla/hAsBA2kGEqaQb0Vp4OTXZ/UJnrfLag0Gay7DT6YQ8FSnbRd7WB09hS6R2Nlr+VgzOsaqrfrU1G6JIwCFiaIDv5aZtoBP5LaGDttV+kgELUs84NVbR1nfNmfyJpfcpCikHJ3lV1dQYr+4+oiFZIWNJAnsOQwiQfQbelJfVbzNJYL5itD4S+p4/bgsKqA9iDo5RUaLBU0uQvUwVngv7RJWSbmsggMKW25wwG7hcK73UWRXP3NcE8Z16tcLPEbRFA5w2OUCf+cB0z6JDgVShEw0TIIwtP9"
        }
      ]
    },
    "imageSKU": {
      "value": "14.04.4-LTS"
    },
    "chef_server_url": {
      "value": "http://my-chef-server/"
    },
    "newStorageAccountName": {
      "value": "tests03"
    },
    "imagePublisher": {
      "value": "Canonical"
    },
    "vmSize": {
      "value": "Standard_A0"
    },
    "runlist": {
      "value": "role[standalone]"
    },
    "imageOffer": {
      "value": "UbuntuServer"
    },
    "validation_key": {
      "value": "PRIVATE-KEY TEXT GOES HERE"
    },
    "validation_client_name": {
      "value": "halfords_test-validator"
    },
    "location": {
      "value": "westeurope"
    },
    "adminUserName": {
      "value": "demouser"
    },
    "publicIPAddressName": {
      "value": "deploy-test-ehw-io"
    },
    "chef_node_name": {
      "value": "demo-node-test-deploys"
    }
  }
}