Failed to update image when upgrade the scale set with another custom image using task Azure VM scale set deployment #11684
Comments
luweilevi
changed the title
|
We have the same issue, it's a BIG pain for us. |
|
@luweilevi the custom image in the error message means a VHD image which has been set using a URI. This task does not support updating a VMSS created using a managed image. |
|
What about this?: Here is a Log: this Task worked well for us last two years, the last successful run log: |
|
@bishal-pdMSFT @damccorm |
|
@chernovol can you please attach debug logs for failed task? Also, it looks like a VM extension is failing to install on VMSS. The task does not install any such extension. But I think when VMSS image gets updated, VMSS tries to re-install all extensions which were previously installed on VMSS. If any such extension is failing, then may be image update itself is reported as failure. Can you please check VMSS in azure portal to see if it is reporting an extension failure. You can remove extension and try the task again. |
|
@bishal-pdMSFT Yes, I can see failure report related to provisioning.
What's the quickest way of generating the debug logs for the failed task? |
|
@hanif-everycity can you delete the extension and retry to task? You can add a pipeline variable with name |
|
@bishal-pdMSFT Please find attached the log for the task. I've masked some sensitive data. 18_Azure VMSS MyCompany-UAT Update image.log With regards to deleting the extension, I removed it from Azure > VMSS > Extensions prior to the above run. |
|
@bishal-pdMSFT |
|
@chernovol the task does not install this `` extension explicitly and hence I can't comment why it is failing. But I have a hunch that updating image on VMSS somehow triggers this extension installation. This could be due to something specific to this VMSS. Can you make a REST call to get more details about this VMSS Another option you can do is to try this image update with a different VMSS (preferably a new VMSS). This will definitely point to some issue with current VMSS. |
|
Thanks @bishal-pdMSFT . I uninstalled the PowerShell DSC extension and restarted the scale set. This resulted in the deployment completing successfully. I reinstated the exact PowerShell DSC extension and script, restarted the scale set and then run the deployment successfully again. It seems to me the extension got stuck somewhere during installation. The script itself works fine on a standalone. |
|
@bishal-pdMSFT We are now seeing the following error:
In Azure, under VMSS > Extension, we are installing the PowerShell DSC extension with the following configuration for LCM. It's defining the credentials for the DSC found in Azure Automation. How do we pass in the Azure Automation key (RegistrationKey)? Where does the extension look for this key?
|
|
Moring @bishal-pdMSFT Can you help us with the above as this is a huge blocker for us |
|
@chernovol I am not right person for PowerShell DSC extension. As per @hanif-everycity 's comment VMSS image update works if PowerShell DSC extension is not installed. The culprit is this extension and you should try to fix it. Unfortunately I can't help with this extension. |
|
We are also experiencing this issue is. |
|
@pl-pack-01 you mean you are also hitting PowerShell DSC extension error? |
|
Hi @bishal-pdMSFT |
|
I apologize for the ambiguous comment. To clarify, we have VMSS nodes that continue to fail with error messages similar to the ones mentioned above. We have gotten the following; Reports of failing provisioning when it appears that provisioning was successful, Reports of failing provisioning with an error message about the DSC Extension timing out, and Reports of failing provisioning with error message that it was unable to connect to the either the license or upgrade store. In each of these cases the nodes all are responsive but a few (between 1 and 4 out of 12) will not complete the DSC Powershell script. Please let me know who can help us with this issue. |
|
@bishal-pdMSFT @damccorm |
|
@chernovol I do not have any contact with PowerShell DSC extension. I did a quick internet search and found this page. You might want to use support section there. |
|
Closing this issue as it is not in VM Scale set deployment task |
|
I disagree with the closure of this ticket without assisting us in finding the group responsible. VMSS is where we are experiencing this issue and I am not experiencing it in other Azure Resources. At the very least this is related to VMSS. It would be different if this were a third party extension totally outside your companies control. |
|
@pl-pack-01 Totaly agree with you! @bishal-pdMSFT @damccorm @vincent1173 @hiyadav @DS-MS @luweilevi @jahsu-MSFT |
|
@chernovol not sure if you are still experiencing this issue but I found mine was due to an Out of Memory Exception that was being masked by the timeout. I was able to refactor my script to be more memory efficient, but another option would be to increase the MaxMemoryPerShellMB setting. Hope this helps. |
|
Has this problem been solved? We are struggling with the same error message. Steps to reproduce
How can we fix that or is it a general problem with the pipeline? |
|
@uF7264 Your matter sounds different from what we were facing. We use a DSC extension to register the DSC pull server on each of the VMs in the scale set. The problem was the extension was failing due to permission issues and this was being reported in the DevOps pipeline during the "Azure VM scale set deployment" step. Instead, we decided to use a PowerShell script with the PS extension to register the DSC pull server on VM startup. |
|
We are no longer experiencing this issue. The changes we made mentioned above resolved it for us. |
When trying to upgrade the scale set with another custom image. The error said "cannot be updated as it uses a platform image", However the scale set was indeed created with a custom image.
Failed to update image for VMSS testvmssapp. Error: VMSS testvmssapp can not be updated as it uses a platform image. Only a VMSS which is currently using a custom image can be updated.
"storageProfile": { "osDisk": { "createOption": "FromImage", "caching": "ReadWrite", "managedDisk": { "storageAccountType": "Standard_LRS" }, "diskSizeGB": 127 }, "imageReference": { "id": "/subscriptions/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx/resourceGroups/testvmssdeployment/providers/Microsoft.Compute/images/MyCustomImage" } },The text was updated successfully, but these errors were encountered: