SSH Between 2 Windows Machines Using pwsh.exe and OpenSSH-Win64 with SSH Cert Auth

[pldmgg]

Goal:

Enter an interactive pwsh.exe session on a (Windows) Remote Host using SSH Public Cert Authentication using BOTH ssh.exe AND pwsh.exe *-PSSession cmdlets

Requirements:

• SSH Client must be on a Windows OS
• SSHD Server must be on a Windows OS
• SSH Cert Authentication must be used
• The Subsystem sshd_config option must be used
• The ForceCommand sshd_config option must NOT be used
• Optionally specify DefaultShell as pwsh.exe under HKLM:\SOFTWARE\OpenSSH if necessary

Environment

Relevant info about ssh client and sshd server machines:

• zerotesting5.zero.lab (192.168.2.13) is the localhost (client) and zerowin16sshb.zero.lab (192.168.2.53) is the Remote Host (server).
• Both are running Windows 2016 Standard
• Both have pwsh 6.1.0-rc1 (and ONLY pwsh 6.1.0-rc1) installed
• Both are running OpenSSH-Win64 7.7.2.0
• ssh-agent Service is running on both machines (sshd -ddd is run manually on the sshd server for each test in order to collect logs)
• Each machine's Signed Host Cert (ssh-rsa-cert-v01@openssh.com AAAA... C:\ProgramData\ssh\ssh_host_rsa_key) is loaded in the ssh-agent for both client and server (not sure if this is necessary)
• zero\zeroadmin's SSH Cert ssh-rsa-cert-v01@openssh.com AAAA... C:\Users\zeroadmin\.ssh\zeroadmin_090618_114109 is loaded in the ssh-agent on the client
• Baseline sshd_config is as follows: https://gist.github.com/pldmgg/2850f834889fc430c08e3fd4e735c3e3

What Works Beyond Any Doubt:

Testing so far proves that SSH Certificate Authentication is configured properly since I can consistently get to an interactive cmd.exe session on the Remote Host via any of the following commands:

ssh zeroadmin@zero@zerowin16sshb

ssh -o "IdentitiesOnly=true" -i "C:\Users\zeroadmin\.ssh\zeroadmin_090618" -i "C:\Users\zeroadmin\.ssh\zeroadmin_090618-cert.pub" zeroadmin@zero@zerowin16sshb

ssh -o "IdentitiesOnly=true" -i "C:\Users\zeroadmin\.ssh\zeroadmin_090618" zeroadmin@zero@zerowin16sshb

For completeness sake, C:\Users\zeroadmin\.ssh\zeroadmin_090618-cert.pub is in this format:

ssh-rsa-cert-v01@openssh.com AAAA...<truncated>...Bg==

What Does Not Work:

• I have not been able to get any of the *-PSSession pwsh.exe cmdlets to work under any circumstances (using SSH Cert Auth).
• The above ssh.exe commands NEVER honor the sshd_config Subsystem setting. What I SHOULD see in sshd logs is the following...

Starting session: subsystem 'powershell' for zero\\zeroadmin from 192.168.2.13 port 51060 id 0

...What I always end up seeing is the following...

Starting session: shell on windows-pty for zero\\zeroadmin from 192.168.2.13 port 50944 id 0

Full sshd logs can be found under the relevant SCENARIO sections below.

Side Note: Using the pwshe.exe *-PSSession cmdlets DOES honor Subsystem (i.e., I see the desired Starting session: subsystem ... in sshd logs), however, they never actually create a PSSession.

The Closest I Got to Satisfying Requirements:

SCENARIO 1:

Config Info (these are the ONLY things that are changed from the above sshd_config between testing scenarios)

• DefaultShell Entry in Registry - NO
• ForceCommand pwsh.exe -NoProfile implemented in sshd_config - YES
• Subsystem powershell C:/symlinks/pwsh.exe -sshs -NoLogo -NoProfile implemented in sshd_config -YES
  (NOTE: C:/symlinks/pwsh.exe is a symlink to C:/Program Files/Powershell/6-preview/pwsh.exe)

Result

• The aforementioned ssh commands place me in pwsh.exe on the Remote Host (SUCCESS)
• *-PSSession cmdlets on the client side fail for some unknown reason (FAILURE)
• Additional Notes:

1. Subsystem is NOT honored when using ssh.exe. Subsystem IS honored when using the *-PSSession cmdlets.
2. Changing ForceCommand pwsh.exe -NoProfile to ForceCommand pwsh.exe -sshs -NoLogo -NoProfile causes ssh.exe to hang on the client side, but things look okay on the sshd side

LOGS:

1. SSHExe_No_DefaultShell_In_Registry: https://gist.github.com/pldmgg/3fa040a73ce6f83385dbc05acb96cf1a
2. New-PSSession_No_DefaultShell_In_Registry: https://gist.github.com/pldmgg/6cf1a2e1ca02ae540062222fc50bbb63

SCENARIO 2:

Config Info (these are the ONLY things that are changed from the above sshd_config between testing scenarios)

• DefaultShell Entry in Registry of C:/Program Files/Powershell/6-preview/pwsh.exe - YES
• ForceCommand implemented in sshd_config - NO
• Subsystem powershell C:/symlinks/pwsh.exe -sshs -NoLogo -NoProfile implemented in sshd_config -YES
  (NOTE: C:/symlinks/pwsh.exe is a symlink to C:/Program Files/Powershell/6-preview/pwsh.exe)

Result

• The aforementioned ssh commands place me in pwsh.exe on the Remote Host (SUCCESS)
• *-PSSession cmdlets on the client side fail for some unknown reason (FAILURE)
• Additional Notes:

1. Subsystem is NOT honored when using ssh.exe. Subsystem IS honored when using the *-PSSession cmdlets.

LOGS:
I can post this as public gists, but long story short, the same problems mentioned in the "What Does Not Work" section above apply.

ALL OTHER SCENARIOS:

Trying config settings other than those outlined in the above 2 Scenarios always resulted in one of the following outcomes:

• Placed me in cmd.exe on the Remote Host
• Prevented me from connecting altogether
• Hung indefinitely on either Client or SSHD side

Attempting to connect using SSH Cert Auth via the *-PSSession cmdlets (i.e. the -HostName, -KeyFilePath, and optionally the -SSHTransport parameters) has never been successful under any circumstances. Specifically I tried the following under all testing scenarios that I attempted...

New-PSSession -HostName zerowin16sshb -UserName zero\zeroadmin -KeyFilePath "C:\Users\zeroadmin\.ssh\zeroadmin_090618-cert.pub"

New-PSSession -HostName zerowin16sshb -UserName zero\zeroadmin -KeyFilePath "C:\Users\zeroadmin\.ssh\zeroadmin_090618"

New-PSSession -HostName zerowin16sshb -UserName zeroadmin -KeyFilePath "C:\Users\zeroadmin\.ssh\zeroadmin_090618-cert.pub"

New-PSSession -HostName zerowin16sshb -UserName zeroadmin -KeyFilePath "C:\Users\zeroadmin\.ssh\zeroadmin_090618"

New-PSSession -HostName zerowin16sshb -UserName zero\zeroadmin -KeyFilePath "C:\Users\zeroadmin\.ssh\zeroadmin_090618-cert.pub" -SSHTransport

New-PSSession -HostName zerowin16sshb -UserName zero\zeroadmin -KeyFilePath "C:\Users\zeroadmin\.ssh\zeroadmin_090618" -SSHTransport

New-PSSession -HostName zerowin16sshb -UserName zeroadmin -KeyFilePath "C:\Users\zeroadmin\.ssh\zeroadmin_090618-cert.pub" -SSHTransport

New-PSSession -HostName zerowin16sshb -UserName zeroadmin -KeyFilePath "C:\Users\zeroadmin\.ssh\zeroadmin_090618" -SSHTransport

Thanks in advance for any help!

[pldmgg]

From /u/ainesophaur on reddit
(https://www.reddit.com/r/PowerShell/comments/9doeqm/has_anyone_successfully_been_able_to_get_to_an/)

"There's a github issue about this and ultimately it was stated that authing by key pair will not grant you a kerberos session.. So you're then unable to do remote powershell using kerberos or negotiate auth. Your options are to either sign in with username and password via ssh (which will give you a kerberos ticket) or you have to use basic auth for remote powershell"

This actually makes a lot of sense to me...is that what's going on here?

EDIT: No it's not...

[pldmgg]

The discussion over here...

PowerShell/Win32-OpenSSH#1246

...has led me to believe that this is not a Kerberos issue.

Here's more information.

Using Password Auth

sshd Logs...
(Problems after line 260)
https://gist.github.com/pldmgg/2b675389975c19d27ea17cc7da14e52e

On the client side, it just hangs...

PS C:\Users\zeroadmin> New-PSSession -HostName zerowin16sshb
zeroadmin@ZERO@zerowin16sshb's password:

# I need the Ctrl-C here, otherwise it just hangs indefinitely

Using SSH (RSA) Cert Auth:

sshd Logs...
(Problems after line 304)
https://gist.github.com/pldmgg/c0172dfbd8a867b5f5c52ae159c14bbd

On the client side...

PS C:\Users\zeroadmin> New-PSSession -HostName zerowin16sshb -UserName zero\zeroadmin -KeyFilePath "C:\Users\zeroadmin\.ssh\zeroadmin_090918_121751-cert.pub"

# I need the Ctrl-C here, otherwise it just hangs indefinitely

@dantraMSFT Sorry to bother you, and I'm not really asking for help with diagnosis, but more of a sanity check:

Have you / anyone you know been able to get pwsh.exe *-PSSession cmdlets to work between two Windows machines via SSH (i.e. -HostName, -KeyFilePath, -SSHTransport parameter set)? If yes, I'll redouble my efforts to try and diagnose my own problem since I'll know of at least one person that's been successful. But right now, I don't know anyone that's been able to make it work (password auth, key auth, or ssh cert auth).

[dantraMSFT]

@pldmgg I've been using Server->Server with OpenSSH regularly for the last few weeks; the target is Server 2016 so I've installed OpenSSH there and also PowerShell RC-1.
Here's an example:

$session = New-PSSession -HostName mytargetserver -UserName testuser -KeyFilePath ~.ssh\testuserprivatekey

On the target system, I have the following configuration:

• sshd_config

Subsystem	powershell c:\powershell\pwsh.exe -sshs -NoLogo -NoProfile
PubkeyAuthentication yes
AuthorizedKeysFile	.ssh/authorized_keys

%username%.ssh

I simply copied the public key for my user to authorized_keys.

I also ran the two permission 'fixing' scripts from the OpenSSH directory (FixHostFilePermissions.ps1 and FixUserFilePermissions.ps1). Like the Linux versions, OpenSSH requires restricted permissions on various files.

Finally, if you installed PowerShell under 'Program Files' on the target system, it will not work with OpenSSH due to a known issue with OpenSSH and subsystem paths that contain spaces. The suggested fix is to create a symlink for pwsh.exe but I found this did not work on my 2016 systems. I ended up creating a directory symlink to the PowerShell Core installation directory. (That's why the above sbsystem references c:\powershell).

[pldmgg]

@dantraMSFT Thanks so much for getting back to me.

Unfortunately, I still haven't had any success getting the *-PSSession cmdlets to work between two Windows 2016 Standard machines. I've tried fresh VMs, different domains, always with the result where pwsh on the client Win16 machine hangs and I need to Ctrl+C.

I'm certain that I've followed all of the aforementioned guidance (symlinks, permissions, etc), but still no dice.

I'm not going to close this issue because I really feel like someone should be able to recreate this issue. But I understand if a contributor wants to close this issue without a solution if the issue can't be recreated.

[dantraMSFT]

Here are a few things to check:
1: Verify the pwsh.exe symlink actually works. Open a command prompt and attempt to run it as you have it defined in sshd_config
2: Let's get the Win32 OpenSSH team involved. There's a few errors in the client side log but I don't know how to interpret them.
Ensure you have logging set up for both client and server as described here: https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps
The I would create an issue here: https://github.com/powershell/Win32-OpenSSH/issues

Feel free to reference me in the issue so I can stay in the loop.
I'm going to be unavailable for a few days so be patient if you're looking for a response from me.

[pldmgg]

HOLY CRAP I figured it out!

My mistake was that I was creating the symlink to pwsh.exe directly as opposed to its parent directory, i.e.

New-Item -ItemType SymbolicLink -Path "C:\symlinks\pwsh.exe" -Target $(Get-Command pwsh).Source

When I made a symlink to pwsh.exe's parent directory (and updated the Subsystem path in sshd_config), the *-PSSession cmdlets started working:

New-Item -ItemType SymbolicLink -Path "C:\pwshRoot" -Target $($(Get-Command pwsh).Source | Split-Path -Parent)

Now we're firing on all cylinders (password auth, key auth, ssh cert auth)

Thanks for your help @dantraMSFT !