PowershellDirect on Windows Server 2016: Invoke-Command -VMName 'blah' fails with Unable to cast object of type 'System.String' to type 'VMState'.

[ayeltsov]

Steps to reproduce

$vmUserPassword = 'password'; #reproes even without correct password
$vmUser = '.\Administrator'; 
$vmMachineName = 'MyLocalVMName';  #this VM needs to exist
$password = ConvertTo-SecureString $vmUserPassword -AsPlainText -Force; 
$cred = New-Object System.Management.Automation.PSCredential ($vmUser, $password); 
Invoke-Command -VMName $vmMachineName -Credential $cred { Write-Output ('Connection'+'Succeeded') }

Expected behavior

ConnectionSucceeded

Actual behavior

Invoke-Command: Unable to cast object of type 'System.String' to type 'VMState'.

Environment data

Name                           Value
----                           -----
PSVersion                      7.1.1
PSEdition                      Core
GitCommitId                    7.1.1
OS                             Microsoft Windows 10.0.14393
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Notes:

• this is about the host, rather than guest guest:
  • Win10 or Windows Server 2019 host + Windows Server 2019 guest => works
  • Windows Server 2016 host + Windows Server 2019 guest => breaks
• this is specific to powershell core
  • the same command works from Powershell ISE 5.1.14393.3866 from either host
• In my actual environment I use powershell SDK from c# app, but thankfully it reproes from regular powershell
• Similar unanswered question on stackoverflow .

[jborean93]

Looking at the code there are 2 places where this could occur

PowerShell/src/System.Management.Automation/engine/remoting/commands/newrunspacecommand.cs

Line 942 in 3c7f4b5

if ((VMState)results[0].Properties["State"].Value != VMState.Running)

and

PowerShell/src/System.Management.Automation/engine/remoting/commands/PushRunspaceCommand.cs

Line 1007 in c7bdb24

if ((VMState)results[0].Properties["State"].Value != VMState.Running)

.

They are both trying to cast the State property from the first output of Get-VM. It seems like the code could be a bit more defensive and try to cast the string to the enum type as that isn't implicitly done leading to the error you get back.

[ayeltsov]

Just to clarify impact and what I'm looking for.

Impact: blocking ability to ship new version of internal tool. We've update the tool's codebase from .NET 4.7 to .NET 5, which necessitated switching to PowerShell Core SDK from older Powershell SDK. Since we have to support Win 2016, Win 2019 and Win 10 OS, we cannot ship the new version of the tool

What I'm looking for: some sort of work around to get past the issue.

Please let me know if I can do anything to help root cause it and look for workaround. Would love to unblock my project as soon as possible.

[jborean93]

I'm not sure you can fix this without changing the PowerShell code. Maybe to start off could you share the output of the following (in both Windows PowerShell and PowerShell):

$vm = @(Get-VM -VMName MyLocalVMName)[0]

$vm.State
$vm.State.GetType().FullName

Comparing that between 2016 and 2019 guests would also help to potentially track down what is happening.

[ayeltsov]

Comparing that between 2016 and 2019 guests
Just to be on the same page - difference is about hosts. Guest is Server 2019 in both cases.

$vm = @(Get-VM)[0]
$vm.State
$vm.State.GetType().FullName

$vm.State is 'Running' in all the cases. Datatype is like this:

Powershell	Server 2019/Windows 10	Server 2016
Powershell ISE	Microsoft.HyperV.PowerShell.VMState	Microsoft.HyperV.PowerShell.VMState
Powershell Core	Microsoft.HyperV.PowerShell.VMState	System.String

[jborean93]

Cool so that code is definitely problematic, I just don't understand why Server 2016 on PowerShell is a string but not for Windows PowerShell. I would have to look at the code to see what it is doing to give you a better answer. In short I think the PowerShell code needs to be a bit more defensive here but it's still an interesting problem.

[ayeltsov]

I'd be happy to try a private fix of Powershell SDK.
Repro-wise I understand it's annoying to try and source Windows 2016 server, especially Hyper-V enabled.

I personally used nested virtualization: Win 10 host -> Win 2016 server [for repro] -> Win 2019 guest, but I guess any VM will do
Not 100% sure, but it may be possible to get nested virtualization enabled Win 2016 sever from Azure.

[SeeminglyScience]

I'm guessing the 2016 version is using implicit remoting (or w/e the automatic wincompat layer is called, I forget) due to the HyperV module not getting the Core tag via Windows Updates. As a workaround try Import-Module HyperV -SkipEditionCheck before running the command.

[jborean93]

Ahh I was assuming guest was the guest OS being targeted and the OS that is running the Invoke-Command was always Windows 10 but that would make more sense.

[ayeltsov]

This workaround worked, thank you!

PS C:\Users\ayeltsov> Import-Module Hyper-V -SkipEditionCheck
PS C:\Users\ayeltsov> (@(Get-VM)[0]).State.GetType().FullName
Microsoft.HyperV.PowerShell.VMState

Let's see if this onion has more layers...

[ayeltsov]

Definitely more layers in this onion.

Same workaround needed for this:

(Get-VM ayeltsov-336442) | Get-VMNetworkAdapter
Get-VMNetworkAdapter: The specified wildcard character pattern is not valid: VirtualMachine (Name = 'ayeltsov-336442') [Id = '69d6d3ef-09d1-4682-8b2a-b6e2cf44c593']

Question: is there a way to hint "-SkipEditionCheck" globally so that I don't change all the scripts that app is using?

In case it matters, I use powershell from c#:

            using (PowerShell powerShell = PowerShell.Create())
            {
                InitialSessionState initialSessionState = InitialSessionState.CreateDefault();
                initialSessionState.ExecutionPolicy = ExecutionPolicy.RemoteSigned;
                powerShell.Runspace = RunspaceFactory.CreateRunspace(initialSessionState);
                powerShell.Runspace.Open();
                <...>
                powerShell.Invoke<PSObject, PSObject>(executionInputCollection, outputCollection, new PSInvocationSettings());
                <...>
            }

[SeeminglyScience]

Question: is there a way to hint "-SkipEditionCheck" globally so that I don't change all the scripts that app is using?

You could try $PSDefaultParameterValues['Import-Module:SkipEditionCheck'] = $true but I doubt autoloading actually calls Import-Module. Even if it does, don't be surprised if some built in modules break in core without the wincompat layer (though tbh about as likely to break in my experience).

If you control the environment you could edit the manifests yourself to add Core under CompatiblePSEditions. Again, ymmv and unlikely to be considered supported.

Last resort if you're desperate would be hooking into command lookup via $ExecutionContext.InvokeCommand.PreCommandLookupAction.

[ayeltsov]

@SeeminglyScience , FYI, the workaround with Import-Module:SkipEditionCheck did actually work (please don't break it in the next SDK :-))
I understand that it's affecting more modules than Hyper-V, but it works for this app.

Just for reference:

                if (IsWindows2016.Value)
                {
                    System.Collections.Hashtable psDefaultParameterValues =
                        (System.Collections.Hashtable)powerShell.Runspace.SessionStateProxy.GetVariable("PSDefaultParameterValues");
                    psDefaultParameterValues["Import-Module:SkipEditionCheck"] = true;
                }

        private static readonly Lazy<bool> IsWindows2016 = new (
            () =>
            {
                // Weirdly, ProductName reg key is easiest way to distinguish Server 2016 and Server 2019.
                // It's not exposed via things like System.Environment.OSVersion
                // Example values: "Windows 10 Enterprise"  "Windows Server 2016 Datacenter"
                string productName = (string)Registry.GetValue(
                    @"HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion",
                    valueName: "ProductName",
                    defaultValue: null);
                return productName?.Contains("Server 2016", StringComparison.OrdinalIgnoreCase) ?? false;
            });

[SeeminglyScience]

@ayeltsov honestly with a C# application, you're probably better off always including that parameter in Core. Unless you're incredibly careful in your usage of returned objects (i.e. never statically typing them), wincompat mode is going to break you regardless.

Another side note, RuntimeInformation.OSDescription can get you there too probably

[SteveL-MSFT]

Unfortunately, it seems that the original issue is "by-design". If a Windows inbox module manifest does not include Core in CompatiblePSEditions, then PowerShell 7 will use implicit remoting with Windows PowerShell to load the module. When this happens, all output is deserialized and the enum becomes a string. We've worked with Windows teams to update their module manifests to include Core where we know it works, but those changes do not get backported to older versions of Windows. -SkipEditionCheck forces Import-Module to load the module directly into the pwsh process. I think the current workaround may be the best solution for now.

[ayeltsov]

Ack. We're ok with workaround, and will probably speed up deprecation of Server 2016 as a supported host.
Not sure what is the protocol for closing the bugs, but I'm Ok with closing