Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't connect through jumpbox #18

Closed
Stephen9s opened this issue May 2, 2019 · 55 comments
Closed

Can't connect through jumpbox #18

Stephen9s opened this issue May 2, 2019 · 55 comments

Comments

@Stephen9s
Copy link

@Stephen9s Stephen9s commented May 2, 2019

Some users may leverage ~/.ssh/config to set up ProxyJump and ProxyCommands to get to their endpoints. Currently, it appears that VSCode Remote only supports a direct connection.

@tralston
Copy link

@tralston tralston commented May 3, 2019

I'm able to connect to a JumpHost endpoint. Mine are configured in my config file. The only thing I had to do was copy my SSH key to the remote endpoint (and jump hosts). For example, if my config was:
local -> jump1 -> jump2 -> server
Then I'd have to run "ssh-copy-id" for jump1, then for jump2, then for server. After that, VSCode connected just fine.

Loading

@Stephen9s
Copy link
Author

@Stephen9s Stephen9s commented May 3, 2019

I am able to do the same, but what I found was that I could do the following:

local -> server = OK
local -> jump1 -> server = able to authenticate, but unable to connect.

I do know that when VSC connects to the remote server, it attempts to install or run a service. My suspicion is that by jumping through from X to Y to Z, the service ends up installing on Y and not X. Perhaps maybe there’s a better way to pivot using JumpHost as you said rather than ProxyCommand.

I’ll give ProxyJump a shot.

Loading

@tralston
Copy link

@tralston tralston commented May 3, 2019

@TheStoof for what it's worth, here's an excerpt from my config file (sanitized):

host jump1-alias
  HostName jump1
  User myUser
  Port 2202

host server-alias
  HostName server
  User myUser
  Port 2202
  ProxyCommand ssh -W %h:%p jump1-alias

Loading

@tralston
Copy link

@tralston tralston commented May 3, 2019

@TheStoof I just attempted the same connection, but from a Windows 10 machine with WSL, and it didn't work (same problem as #34). It did work though from my Mac.

Loading

@msalvaris
Copy link

@msalvaris msalvaris commented May 7, 2019

Yea, tried various options and nothing succesful to far. I was able to connect once by creating a tunnel in WSL and pointing the Hostname and Port to the tunnel but wasn't able to repeat.

Loading

@pd93
Copy link

@pd93 pd93 commented May 9, 2019

Ran into this issue while trialing Remote SSH as an alternative to local development. Connecting from a Windows 10 machine to a remote box works, but only when connecting directly. We cannot connect to any hosts via a jumpbox. In other words:

  • local -> jumpbox works
  • local -> jumpbox -> remote doesn't work

Here's my (sanitized) config:

Host jumpbox
    HostName    12.34.56.78
    Port        22
    User        user

Host remote
    Hostname    172.16.0.1
    Port        22
    User        user
    ProxyJump   jumpbox

There is no way for us to SSH into a remote machine without tunneling through a jumpbox, so this functionality it absolutely necessary for us to use Remote - SSH.

Loading

@LeuisKen
Copy link

@LeuisKen LeuisKen commented May 10, 2019

I'm trying to solve this issue by writing a new RemoteAuthorityResolver, which the Remote SSH and Remote Container based on. But there is no document about this API...

Loading

@suanrong
Copy link

@suanrong suanrong commented May 20, 2019

Anyone solve this issue?

Loading

@msalvaris
Copy link

@msalvaris msalvaris commented May 20, 2019

@suanrong

Anyone solve this issue?

I was able to overcome the issue by using this method
#117

I create the proxy in wsl and simply make sure that my ssh config points to localhost and the correct port.

Loading

@pd93
Copy link

@pd93 pd93 commented May 30, 2019

I decided to try SSHing (with the same config as above) to my remote boxes via Powershell today. This should use the same version of SSH as the Remote - SSH extension (rather than the WSL version of SSH).

Unsurprisingly, I got exactly the same result, but a better error message

  • local -> jumpbox works
  • local -> jumpbox -> remote fails with the error:
CreateProcessW failed error:2
posix_spawn: No such file or directory

A quick search of the error found PowerShell/Win32-OpenSSH#1185.
Not 100% sure if related, but that issue was fixed in Jan which implies that if that issue is the problem, this would be fixed in 1903. Unfortunately, we're still on 1809 at work, so I can't test.

A bit more searching also found PowerShell/Win32-OpenSSH#1172. This issue seems more relevant than 1185, but is still open 😢

Loading

@roblourens roblourens changed the title Offer ability to tunnel through to endpoint Can't connect through jumpbox May 30, 2019
@axiqia
Copy link

@axiqia axiqia commented May 31, 2019

I have copyed my SSH key to the remote endpoint and jump hosts as @tralston suggested. But I still met the issue like @pd93 . I added more information.

My config:

Host jumpbox
    HostName example.edu.cn
    User U
    Port xxx
    IdentityFile xxx.id_rsa


Host remote
    HostName remote
    User U
    Port xx
    ProxyCommand ssh -q -W %h:%p jumpbox

On Git Bash(windows)

$ ssh -V                                                                         
OpenSSH_7.9p1, OpenSSL 1.1.1a  20 Nov 2018
  • local -> jumpbox works
ssh jumpbox
  • local -> jumpbox -> remote works
ssh remote

On Powershell:

ssh -V
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
  • local -> jumpbox works
ssh jumpbox
  • local -> jumpbox -> remote fails with the error:
ssh remote
CreateProcessW failed error:2
posix_spawn: No such file or directory

On VS Code:

  • local -> jumpbox works
  • local -> jumpbox -> remote fails with the error:
Picking SSH host
Selected remote
Confirming that remote is a valid reachable host
Running type "C:\Users\xxx\AppData\Local\Temp\vscode-linux-multi-line-command-remote.sh" | ssh  "remote" bash to confirm the host platform
> 
> CreateProcessW failed error:2
> posix_spawn: No such file or directory
> 
"uname" terminal command done
remote: unreachable or not Linux x86_64. (posix_spawn: No such file or directory)

Loading

@johnymachine
Copy link

@johnymachine johnymachine commented Jun 7, 2019

Maybe this could help:

ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe jumphost netcat -w 120 %h %p

https://serverfault.com/questions/956613/windows-10-ssh-proxycommand-posix-spawn-no-such-file-or-directory

It worked as expected on Windows 10 1809 but there is probably bug in 1903 and this worked for me.

Loading

@davesdig
Copy link

@davesdig davesdig commented Jun 11, 2019

Seeing same problem here when trying to connect to Centos 7 via a Centos 7 jump box from a mac. Tried several forms of ProxyCommand and ProxyJump. Can connect thru proxy to target from command line so the configuration should be good. Keys are set up on both jump box and target box.

Code Version: 1.35.0
Extension Version: 0.42.0

Loading

@Higher-Stark
Copy link

@Higher-Stark Higher-Stark commented Jun 11, 2019

I encountered CreateProcessW failed error:2 posix_spawn: No such file or directory on my Windows 10 1903. The ProxyCommand in my config is ProxyCommand ssh jumphost -W %h:%p.
Then I change the command to ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe jumphost -W %h:%p, and it just works.

@johnymachine I tried yours, but I got bash: netcat: command not found.
I am not going to look into the error, I'd appreciate it if you let me know the why this happens.

OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
Windows 10 - 1903

Loading

@johnymachine
Copy link

@johnymachine johnymachine commented Jun 11, 2019

I encountered CreateProcessW failed error:2 posix_spawn: No such file or directory on my Windows 10 1903. The ProxyCommand in my config is ProxyCommand ssh jumphost -W %h:%p.
Then I change the command to ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe jumphost -W %h:%p, and it just works.

@johnymachine I tried yours, but I got bash: netcat: command not found.
I am not going to look into the error, I'd appreciate it if you let me know the why this happens.

OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
Windows 10 - 1903

Sorry my bad, I probably copied bad command.

My config is exactly this and it works:
ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p -q user@ipaddr

Loading

@axiqia
Copy link

@axiqia axiqia commented Jun 11, 2019

ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -W %h:%p -q user@ipaddr
@johnymachine Thank you. That works for me on my windows platform.

Loading

@asmitde
Copy link

@asmitde asmitde commented Jun 16, 2019

@johnymachine Thanks for the solution! Do you know how to how to run the proxy ssh process in the background? There's a blank ssh process window that stays open as long as the connection is active.

Loading

@johnymachine
Copy link

@johnymachine johnymachine commented Jun 16, 2019

Loading

@pd93
Copy link

@pd93 pd93 commented Jul 1, 2019

Thought I'd post a little update to this issue for those who run into it.

The root cause of this issue is a bug with OpenSSH for windows. However, this has now been patched in the latest release (v8.0.0.0) (v8.1.0.0).

Updates:

  • Unfortunately, this likely won't get built into Windows until 1909 comes around.
  • Seems like this didn't get into 1909. Waiting on news for when this patch might be available in a Windows release.
  • We can expect to see the latest version of OpenSSH (v8.1) included in Windows 10 for the July 2020 release (see comment). However, in the mean time, you can manually patch it yourself!
  • This comment has now been further clarified. This fix was not included in the Windows 2004 update, but will be released this fall.
  • MSFT has created a pinned issue explaining why OpenSSH was not included in the 20H2 release. It now looks to be released in a servicing update. Please see the linked issue for further info.

Manual Fix

  • Download the v8.1.0.0 release .zip.
  • Extract the .zip file
  • Copy and replace the extracted ssh.exe binary into your C:\Windows\System32\OpenSSH folder.
  • Done! You can now use "Remote Development - SSH" with ProxyJump in the expected way.

If you get a permissions error then this may be because there is system protection in place on the existing ssh.exe binary. This can be removed by setting yourself as the owner of the file (requires admin) and granting yourself "full control". To do this, follow these steps:

  • Right-click on the existing file C:/Windows/System32/OpenSSH/ssh.exe and select Properties.
  • Open the Security tab and click on Advanced.
  • Click the button to change the owner and change it from TrustedOwner to your Windows username.
  • Type your username and click the Check Names button (This should autofill with your full username/email).
  • Apply the settings and then close the properties window and reopen it.
  • Navigate back to the Security tab and click on Edit.
  • Select the Users group and give the group Full Control.
  • Click Apply and exit.
  • Now go back and try patching the file again.

Loading

@JustinGrote
Copy link

@JustinGrote JustinGrote commented Mar 4, 2020

If I turn off "Use Local Server" it gets further and installs the server and starts it, and I can see it running as node on the remote system, however, it just hangs here and never connects. If I turn off "Use Local Server" it doesn't get past the remote port parsing again.

vscode-server already installed. Skipping download...
vscode-server with 3943d614597f9f7c384846613d417e97ea6d5ab4 is already running.
61359013d8ae: start
agentPort==2728==
osReleaseId==windows==
osVersion==6.3.9600==
arch==x64==
platform==windows==
61359013d8ae: end
Install script is 680, watching sshd parent 3396

Loading

@roblourens
Copy link
Member

@roblourens roblourens commented Mar 4, 2020

@JustinGrote I honestly don't know about your scenario - could you open a new issue with the full log, and some more details about what you're doing?

Loading

@JustinGrote
Copy link

@JustinGrote JustinGrote commented Mar 4, 2020

@roblourens I just found my problem is #25, I'll continue it there. Doesn't appear to have anything to do with the port forwarding specifically, just coincidence.

Loading

@bagajjal
Copy link

@bagajjal bagajjal commented Mar 4, 2020

@JustinGrote , @roblourens - If it's a OpenSSH issue then please create an issue here

Loading

@bagajjal
Copy link

@bagajjal bagajjal commented Mar 4, 2020

Please provide the detailed repro steps isolating the vscode environment (Just using ssh client and sshd server).

Loading

@JustinGrote
Copy link

@JustinGrote JustinGrote commented Mar 4, 2020

@bagajjal My Particular Issue apparently unrelated to this issue is opened as #2475. According to @roblourens it's been fixed downstream and will be in next release, so further discussion of my issue should go there :)

Loading

@thohe973
Copy link

@thohe973 thohe973 commented Mar 20, 2020

I got it working by also sending my ssh key to the proxy.

Host random_name
  HostName <host name>
  Port 22
  User <host user>
  IdentityFile <private ssh key>
  ProxyCommand C:\Windows\System32\OpenSSH\ssh.exe -i <private ssh key> -W %h:%p -q <proxy user>@<proxy host>

Loading

@veekayr
Copy link

@veekayr veekayr commented Apr 11, 2020

Below command is working

ProxyCommand C:\\Windows\\System32\\OpenSSH\\ssh.exe -i <ssh-key> -q -W %h:%p jumpbox

Loading

@holoword
Copy link

@holoword holoword commented Apr 14, 2020

if ProxyCommand C:\\Windows\\System32\\OpenSSH\\ssh.exe -i <ssh-key> -q -W %h:%p jumpbox doesn't work, Just use another ssh client. Just like git bin bash ssh client
set Remote.ssh.path in settings as like this
C:\Program Files\Git\usr\bin\ssh.exe or any other ssh client you have

Loading

@Querela
Copy link

@Querela Querela commented May 4, 2020

For me worked the following: [on Win10, w/ w/o WSL]


As for my ssh-key login:
For both hosts, I needed to specify my id_rsa_vscode file. That I needed to do regardless of proxying or not.

Host *
   IdentityFile C:\Users\<username>\.ssh\id_rsa_vscode

[...]

And I needed to copy my keys with ssh-copy-id to the remote hosts. I also needed to specify -f because I used different id_rsa files for normal ssh login in WSL and for VSCode, and without -f it would not copy the second VSCode ssh key. (That was neccessary because VSCode could not work with ssh-agent or password protected ssh identity files - as far as I know - without always typing the remote ssh passwords on each (re-)connect.)

Loading

@eromoe
Copy link

@eromoe eromoe commented Jun 1, 2020

I patched windows 10 before.
After upgrading to Windows 10 2004(20H1) .

C:\Users\xxxx>ssh -vvv dc42
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
debug1: Reading configuration data C:\\Users\\xxxx/.ssh/config
debug1: C:\\Users\\xxxx/.ssh/config line 1: Applying options for *
debug1: C:\\Users\\xxxx/.ssh/config line 152: Applying options for dc42
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug1: Executing proxy command: exec ssh dc_entry -W dc42:22
debug3: spawning "C:\\Windows\\System32\\OpenSSH\\ssh dc_entry -W dc42:22"
CreateProcessW failed error:2
posix_spawn: No such file or directory

Is this a new but or not ?

Loading

@pd93
Copy link

@pd93 pd93 commented Jun 1, 2020

@eromoe Unfortunately, that's still version 7.7 which does not contain the fix. You'll need to manually update to v8.0+ or use one of the other workarounds mentioned above.

It has been confirmed in another issue that v8.1 will be released in an update later this year, but is not included in Windows 10 2004 (20H1):

v8.1 goes into the next release (not into win10 version 2004) available during fall time.

Loading

@anji993
Copy link

@anji993 anji993 commented Jun 7, 2020

@eromoe @pd93 Did you solve the problem after upgrading to Windows 10 2004 by patching it again. I meet the same problem after I upgrade to Windows 10 2004. But patching manually updating to v8.0+ does not solve this problem.

Loading

@eromoe
Copy link

@eromoe eromoe commented Jun 8, 2020

@anji993 Patching does sovle my problem , may be something wrong with previllage ?

Loading

@susguzman
Copy link

@susguzman susguzman commented Jun 17, 2020

For me worked the following: [on Win10, w/ w/o WSL]

As for my ssh-key login:
For both hosts, I needed to specify my id_rsa_vscode file. That I needed to do regardless of proxying or not.

Host *
   IdentityFile C:\Users\<username>\.ssh\id_rsa_vscode

[...]

And I needed to copy my keys with ssh-copy-id to the remote hosts. I also needed to specify -f because I used different id_rsa files for normal ssh login in WSL and for VSCode, and without -f it would not copy the second VSCode ssh key. (That was neccessary because VSCode could not work with ssh-agent or password protected ssh identity files - as far as I know - without always typing the remote ssh passwords on each (re-)connect.)

It worked for me :)
Thank you

Loading

@nicolaipre
Copy link

@nicolaipre nicolaipre commented Sep 15, 2020

FWIW: For anyone wanting password authentication only, this config works on Linux for jump-hosts:

Host <alias>
    HostName <hostname/ip-addr>
    Port 22
    User <user>
    ProxyCommand ssh -W %h:%p -q <user>@<jump-host-ip>

Loading

@tysonite
Copy link

@tysonite tysonite commented Oct 24, 2020

Have anyone seen #3857 that with jump server too?

Loading

@pd93
Copy link

@pd93 pd93 commented Oct 29, 2020

A quick update for anyone following this issue. MSFT currently have a pinned issue explaining why OpenSSH was not included in the 20H2 release. I've updated my workaround comment above to reflect this.

Loading

@plevold
Copy link

@plevold plevold commented Dec 11, 2020

The manual fix by @pd93 worked like a charm for me, but a Windows update seems to have overwritten the updated ssh.exe. Rather than potentially having to replace the executable after future updates I simply extracted the zip to a different location and added it to the systems PATH. My steps:

  • Download OpenSSH v8.1.0.0 .zip
  • Extract to a suitable location, e.g. C:\bin\OpenSSH-Win64
  • Open Edit the system environmental variables (found in the control panel or start menu)
  • Edit the Path variable and add C:\bin\OpenSSH-Win64
  • Make sure that the new entry is above the entry %SYSTEMROOT%\System32\OpenSSH\
  • Restart VSCode

Note that I had to edit PATH for the whole system, not my user. In the latter case the entry will be added after the system version giving it precedence.

Beware that sometime in the future this solution could cause problems if the system OpenSSH version is updated to a more recent version than the manually installed one...

Loading

@matijagrcic
Copy link

@matijagrcic matijagrcic commented Dec 15, 2020

Manual Fix

  • Download the v8.1.0.0 release .zip.
  • Extract the .zip file
  • Copy and replace the extracted ssh.exe binary into your C:\Windows\System32\OpenSSH folder.
  • Done! You can now use "Remote Development - SSH" with ProxyJump in the expected way.

If you get a permissions error then this may be because there is system protection in place on the existing ssh.exe binary. This can be removed by setting yourself as the owner of the file (requires admin) and granting yourself "full control". To do this, follow these steps:

  • Right-click on the existing file C:/Windows/System32/OpenSSH/ssh.exe and select Properties.
  • Open the Security tab and click on Advanced.
  • Click the button to change the owner and change it from TrustedOwner to your Windows username.
  • Type your username and click the Check Names button (This should autofill with your full username/email).
  • Apply the settings and then close the properties window and reopen it.
  • Navigate back to the Security tab and click on Edit.
  • Select the Users group and give the group Full Control.
  • Click Apply and exit.
  • Now go back and try patching the file again.

Worked without any issues. I've used MobaXterm for a long time to first setup the tunnel and then just connect to the destination, but now it's not needed anymore with ProxyJump working correctly.

Loading

@AdorableYoyo
Copy link

@AdorableYoyo AdorableYoyo commented Aug 16, 2021

For me worked the following: [on Win10, w/ w/o WSL]

As for my ssh-key login:
For both hosts, I needed to specify my id_rsa_vscode file. That I needed to do regardless of proxying or not.

Host *
   IdentityFile C:\Users\<username>\.ssh\id_rsa_vscode

[...]

And I needed to copy my keys with ssh-copy-id to the remote hosts. I also needed to specify -f because I used different id_rsa files for normal ssh login in WSL and for VSCode, and without -f it would not copy the second VSCode ssh key. (That was neccessary because VSCode could not work with ssh-agent or password protected ssh identity files - as far as I know - without always typing the remote ssh passwords on each (re-)connect.)

life saving!!! I had to log in to appreciate your share! Got mine solved! thanks!

Loading

@pd93
Copy link

@pd93 pd93 commented Aug 16, 2021

@roblourens I believe this can be closed now. This was never a VSCode issue and it should now be resolved anyway as a newer (and working) version of OpenSSH is now included in the latest version of Windows.

Anyone on older versions of Windows (and therefore OpenSSH) can follow the manual steps I outlined earlier in this issue in order to update.

Loading

@roblourens
Copy link
Member

@roblourens roblourens commented Aug 23, 2021

Thanks for pointing that out @pd93

Loading

@roblourens roblourens closed this Aug 23, 2021
@github-actions github-actions bot locked and limited conversation to collaborators Oct 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet