"Get-AzVM -Status" sometimes fails to get power state for Linux VMs

[o-l-a-v]

Description

We have a runbook that starts and stops VMs based on tags and current time. When we started using some Linux VMs (Cent OS 7) we some times got an error because Az.Compute Get-AzVM -Status failed to obtain power state.

We until recently used AzureRM, but last week updated the runbook to use latest Az modules (Az.Accounts, Az.Compute, Az.Resources) instead, in hope of that fixing the problem. But it did not.

Most of the times, like 98%, Get-AzVM -Status manages to get power state, but some times it does not. I even try twice in the runbook, which seems to indicate this might be an error with ARM backend rather than Az.Compute, but I don't really know where this should be reported. So I start here.

Steps to reproduce

$VMs = [array](
    (Get-AzVM).ForEach{
        $null = Add-Member -InputObject $_ -MemberType 'NoteProperty' -Name 'Statuses' -Value (
            (Get-AzVM -ResourceGroupName $_.'ResourceGroupName' -Name $_.'Name' -Status).'Statuses'
        )
        $_
    }
)

$VMs | Select-Object -Property 'Name','Location',@{'Name'='PowerState';'Expression'={$_.'Statuses'.Where{$_.'Code' -like 'PowerState/*'}.'DisplayStatus'}},@{'Name'='OSType';'Expression'={if($_.'OSProfile'.'WindowsConfiguration'.'ProvisionVMAgent'){'Windows'}else{'Linux'}}}

Environment data

Because this is in a Automation Account, I ran following to get env info.

$PSVersionTable

Import-Module -Name 'Az.Accounts','Az.Compute','Az.Resources'
Get-Module -Name 'Az.*' | Select-Object -Property 'Name','Version'

Output

Name                           Value                                                                                    
----                           -----                                                                                    
PSVersion                      5.1.15063.726                                                                            
PSEdition                      Desktop                                                                                  
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                  
BuildVersion                   10.0.15063.726                                                                           
CLRVersion                     4.0.30319.42000                                                                          
WSManStackVersion              3.0                                                                                      
PSRemotingProtocolVersion      2.3                                                                                      
SerializationVersion           1.1.0.1                                                                                  


Name        Version
----        -------
Az.Accounts 1.9.4  
Az.Compute  4.4.0  
.0  
Az.Resources 2.5.1  

Debug output

Not feasible, can't reproduce the results because it seems to happen at a specific time or when VM is in a specific state. Running the same runbook 30 minutes later, it will not encounter the same issue.

Error output

No error message is given. The object I get returned with Get-AzVM -Status simply has no value for power state.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Drewm3, @avirishuv.

[dingmeng-xue]

Compute team, please help to look into this question.

[amjads1]

Looking into this issue.

[amjads1]

Assigned to Avi.

[o-l-a-v]

@dingmeng-xue
@amjads1

Any updates to share here?

[amjads1]

@o-l-a-v - Avi is looking into the issue and you should expect an update sometime next week.

[o-l-a-v]

@amjads1

Should I rather create a Azure support ticket?

[avirishuv]

@o-l-a-v thanks for reporting this. Is it possible to check that when the power state is missing in the result of the Azure PowerShell query, at that time the direct GET API call on the VM also shows missing power state?
Sometimes, Power state may be missing for VMs due to reliability issues in the implementation, which we are continuing to improve upon.
If you would like more details of the specific incident that you have experienced, feel free to open a support ticket as well.

[o-l-a-v]

@o-l-a-v thanks for reporting this. Is it possible to check that when the power state is missing in the result of the Azure PowerShell query, at that time the direct GET API call on the VM also shows missing power state?
Sometimes, Power state may be missing for VMs due to reliability issues in the implementation, which we are continuing to improve upon.
If you would like more details of the specific incident that you have experienced, feel free to open a support ticket as well.

Sorry for late reply. It happens randomly, and next run 30 minutes later it usually works again. So not feasible to sit on standby until it happens again. I could maybe add the GET API call to the runbook using token from Get-AzContext, hmm. But don't think I'll get an aproval on doing that for the sake of troubleshooting only.

We've now just scripted a workaround, so now I will not even get alerted when it happens. The workaround is to just send Start or Stop cmdlet even if we fail to get power state for a VM. If the VM already runs and you invoke Start-AzVM, nothing happens, so seems like a safe workaround.

Edit: PowerShell to get power state with Rest API

# Assets
$ResourceGroupName = ''
$ResourceName = ''

# Get
$AzContext = Get-AzContext
$ArmToken = [Microsoft.Azure.Commands.Common.Authentication.AzureSession]::Instance.AuthenticationFactory.Authenticate(
    $AzContext.'Account',
    $AzContext.'Environment',
    $AzContext.'Tenant'.'Id',
    $null,
    [Microsoft.Azure.Commands.Common.Authentication.ShowDialog]::Never,
    $null,
    'https://management.azure.com/'
)
$Uri = 'https://management.azure.com/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.Compute/virtualMachines/{2}/instanceView?api-version=2020-06-01' -f $AzContext.'Subscription'.'Id', $ResourceGroupName, $ResourceName
$Headers = [hashtable]@{'Authorization'='Bearer {0}'-f$ArmToken.'AccessToken'}
(Invoke-RestMethod -Method 'Get' -Headers $Headers -Uri $Uri).'Statuses' | Format-List

[avirishuv]

@o-l-a-v thanks for the update about your workaround. We will be updating the public doc to reflect that sometimes the power state may be missing due to reliability.

[avirishuv]

Apologies for the delay due to holidays, still working on the documentation for this, will update here once done.

[avirishuv]

quick update: reviewing the doc update.

[avirishuv]

The document is updated: https://docs.microsoft.com/en-us/azure/virtual-machines/states-billing

Closing the issue now, please feel free to reopen if you have any additional questions on this topic.

[philhines]

@avirishuv : I am noticing that this issue hasn't been fixed yet. There has been more than enough time to not just document a workaround, but actually fix the bug. Please fix it. Thanks!
This is affecting me as well!

[Drewm3]

@philhines, the return of the power state is not expected to be 100%. There are a small number of cases where this is expected to be null due to various possible failures. The failure rate should be pretty low, though. If you have a scenario where this is returning null consistently, then please provide the detailed method to show the consistent reproduction of this issue.