[Remote-SSH Bug]: Connection hangs at "checking for a running server"

[chuckds]

Is there an existing issue for this bug?

• I have searched the existing issues

Required Troubleshooting Steps

• I have followed these troubleshooting steps
• I have tried both values of the remote.SSH.useLocalServer setting

Connect Locally

It connects successfully

->

No response

Expected Behavior

On connecting to the remote host I expect VSCode to complete that connection and not hang indefinitely.

Steps To Reproduce

Requirements:

1. A host with a bad NFS mount - one that means that any attempt to access files on it results in the accessing process to hang. For illustrative purposes lets call this /bad/mount/path
2. On that host a command needs to have been run with an argument which is a globed path on the bad mount e.g. ls /bad/mount/path/*. Given the requirement above this command will hang - remaining in the output of ps ax.
3. Attempt to connect to this server using VSCode Remote SSH.
4. The set up will get as far as the following line and then hang:

Checking /<homedir>/.vscode-server/.e2816fe719a4026ffa1ee0189dc89bdfdbafb164.log and /<homedir>/.vscode-server/.e2816fe719a4026ffa1ee0189dc89bdfdbafb164.pid for a running server

If you don't have a server with a broken NFS mount to hand it might be possible to produce an issue where the remote connection has a very big delay (though not an indefinite hang) with the following:

1. On a host create the file noop.sh with the following contents:

kill -STOP $$

2. Run bash noop.sh "/*/*/*/*/*/*/*" (don't skip the quotes). This will run bash which will immediately stop itself. This process, along with its argument, will appear in the output of ps ax
3. Attempt to connect to this server using VSCode Remote SSH.
4. Assuming the host has quite a lot of files accessible in its filesystem (such that expanding the wildcard /*/*/*/*/*/*/*) then there will be a delay (vs if bash noop.sh... wasn't run) when connecting to this host using VSCode Remote SSH and the delay will be after the following line is printed in the output window:

Checking /<homedir>/.vscode-server/.e2816fe719a4026ffa1ee0189dc89bdfdbafb164.log and /<homedir>/.vscode-server/.e2816fe719a4026ffa1ee0189dc89bdfdbafb164.pid for a running server

Remote-SSH Log

Remote-SSH Log

[23:27:19.166] Log Level: 2
[23:27:19.167] SSH Resolver called for "ssh-remote+<host>", attempt 1
[23:27:19.167] "remote.SSH.useLocalServer": true
[23:27:19.168] "remote.SSH.path": undefined
[23:27:19.168] "remote.SSH.configFile": undefined
[23:27:19.168] "remote.SSH.useFlock": false
[23:27:19.168] "remote.SSH.lockfilesInTmp": false
[23:27:19.168] "remote.SSH.localServerDownload": auto
[23:27:19.168] "remote.SSH.remoteServerListenOnSocket": false
[23:27:19.168] "remote.SSH.showLoginTerminal": false
[23:27:19.168] "remote.SSH.defaultExtensions": []
[23:27:19.168] "remote.SSH.loglevel": 2
[23:27:19.168] "remote.SSH.enableDynamicForwarding": true
[23:27:19.168] "remote.SSH.enableRemoteCommand": false
[23:27:19.168] "remote.SSH.serverPickPortsFromRange": {}
[23:27:19.168] "remote.SSH.serverInstallPath": {}
[23:27:19.170] VS Code version: 1.75.0
[23:27:19.170] Remote-SSH version: remote-ssh@0.96.0
[23:27:19.170] darwin arm64
[23:27:19.171] SSH Resolver called for host: <host>
[23:27:19.171] Setting up SSH remote "<host>"
[23:27:19.172] Acquiring local install lock: /var/folders/tr/5_z7mhdx4vjghlm9dqp9083c0000gn/T/vscode-remote-ssh-a63a68a7-install.lock
[23:27:19.173] Looking for existing server data file at /Users/<user>/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-a63a68a7-e2816fe719a4026ffa1ee0189dc89bdfdbafb164-0.96.0/data.json
[23:27:19.173] Using commit id "e2816fe719a4026ffa1ee0189dc89bdfdbafb164" and quality "stable" for server
[23:27:19.175] Install and start server if needed
[23:27:19.176] PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/<user>/.cargo/bin
[23:27:19.176] Checking ssh with "ssh -V"
[23:27:19.191] > OpenSSH_9.0p1, LibreSSL 3.3.6

[23:27:19.192] askpass server listening on /var/folders/tr/5_z7mhdx4vjghlm9dqp9083c0000gn/T/vscode-ssh-askpass-5cdd6387f1e1b58595162321b66f509da34593db.sock
[23:27:19.192] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/tr/5_z7mhdx4vjghlm9dqp9083c0000gn/T/vscode-ssh-askpass-4c1da9e21ae8977420c336d04337b1642f609825.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","63937","-o","ConnectTimeout=15","<host>","bash"],"serverDataFolderName":".vscode-server","dataFilePath":"/Users/<user>/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-a63a68a7-e2816fe719a4026ffa1ee0189dc89bdfdbafb164-0.96.0/data.json"}
[23:27:19.192] Local server env: {"SSH_AUTH_SOCK":"/private/tmp/com.apple.launchd.ZpkLcSZDyv/Listeners","SHELL":"/bin/zsh","DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/<user>/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/Users/<user>/.vscode/extensions/ms-vscode-remote.remote-ssh-0.96.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/tr/5_z7mhdx4vjghlm9dqp9083c0000gn/T/vscode-ssh-askpass-5cdd6387f1e1b58595162321b66f509da34593db.sock"}
[23:27:19.193] Spawned 40007
[23:27:19.259] > local-server-1> Running ssh connection command: "-v -T -D 63937 -o ConnectTimeout=15 <host> bash"
[23:27:19.261] > local-server-1> Spawned ssh, pid=40012
[23:27:19.269] stderr> OpenSSH_9.0p1, LibreSSL 3.3.6
[23:27:20.228] stderr> debug1: Server host key: ssh-ed25519 SHA256:ZF29b717uvwlj/HlhV5KQTq1/egBI5JRLpnz2witCKc
[...snipped MOTD...]
[23:27:20.911] stderr>      
[23:27:21.375] stderr> Authenticated to <host> ([<ip>]:22) using "publickey".
[23:27:22.214] > ready: e2995af2ee70
[23:27:22.423] > Linux 3.10.0-1160.71.1.el7.x86_64 #1 SMP Wed Jun 15 08:55:08 UTC 2022
[23:27:22.424] Platform: linux
[23:27:22.632] > /bin/bash
[23:27:22.632] Shell: bash
[23:27:22.632] Creating bash subshell inside "bash"
[23:27:22.847] > bash version:  4.2.46(2)-release
[23:27:22.912] > bash version:  4.2.46(2)-release
[23:27:22.947] > bash version:  4.2.46(2)-release
[23:27:23.002] > bash version:  4.2.46(2)-release
[23:27:23.171] > bash version:  4.2.46(2)-release
> e2995af2ee70: running
> Acquiring lock on /<homedir>/.vscode-server/bin/e2816fe719a4026ffa1ee0189dc89bdfdbafb164/vscode-remote-lock.<user>.e2816fe719a4026ffa1ee0189dc89bdfdbafb164
> \ln /<homedir>/.vscode-server/bin/e2816fe719a4026ffa1ee0189dc89bdfdbafb164/vscode-remote-lock.<user>.e2816fe719a4026ffa1ee0189dc89bdfdbafb164.target /<homedir>/.vscode-server/bin/e2816fe719a4026ffa1ee0189dc89bdfdbafb164/vscode-remote-lock.<user>.e2816fe719a4026ffa1ee0189dc89bdfdbafb164
> Installing to /<homedir>/.vscode-server/bin/e2816fe719a4026ffa1ee0189dc89bdfdbafb164...
> e2995af2ee70%%1%%
> Downloading with wget
[23:27:23.538] > Download complete
[23:27:23.543] > e2995af2ee70%%2%%
> tar --version:
[23:27:23.549] > tar (GNU tar) 1.26
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> 
> Written by John Gilmore and Jay Fenlason.
[23:27:24.892] > Checking /<homedir>/.vscode-server/.e2816fe719a4026ffa1ee0189dc89bdfdbafb164.log and /<homedir>/.vscode-server/.e2816fe719a4026ffa1ee0189dc89bdfdbafb164.pid for a running server
...
<hang>

Anything else?

I think the issue is in ms-vscode-remote.remote-ssh-0.96.0/out/extension.js, specifically with lack of quoting around the ALL_PROCS variable in the bash script:

[...]
ALL_PROCS=`ps ax`
fi
RUNNING="`echo $ALL_PROCS | grep $VSCH_SERVER_SCRIPT | grep -v grep | wc -l | tr -d '[:space:]'`"
[...]

Because ALL_PROCS isn't quoted every token in it is processed by the shell and if it is a wildcard it will be expanded by the shell.

Because ps ax outputs all running process and their arguments, if any of those arguments are wildcard paths that either hang (as in my case) or take a long time to expand (in the case of /*/*/*/*/*) then the remote connection stalls when the subsequent echo call triggers them to be expanded.

If ALL_PROCS can be quoted then this skips any wildcard expanding by the shell and might solve the issue.
But also since echo $ALL_PROCS will only ever be 1 line long the pipeline could probably be reworked.

I raised this issue over a year ago (see issue #5799) but I couldn't see a way to reopen that.
At the time I didn't get to the root cause of the problem and had a workaround and so didn't pursue the problem further.

[chuckds]

Any update? If any more information is needed let me know.
I think the are a few ways it could be fixed. Adding quoting might be tricky due to the need to escape them. Another option might be to ignore the check if the pid file doesn't exist and assume the server isn't running in that case.

[eleanorjboyd]

will review- sorry for the delay!

[chuckds]

Hi Fabian, With the caveat that there are likely other issues that might cause the same symptom: If you have hit this particular issue and you are sure that the vscode server is not running on the host you are trying to connect to (check ps ax output) then the workaround that I use is to create the pid file that vscode tries to look for i.e.: ``` echo "123445667890" > /<homedir>/.vscode-server/.e2816fe719a4026ffa1ee0189dc89bdfdbafb164.pid ``` Where the path you echo that to is the second path that is output in the log output i.e. the 2nd file in the following log: ``` Checking /<homedir>/.vscode-server/.e2816fe719a4026ffa1ee0189dc89bdfdbafb164.log and /<homedir>/.vscode-server/.e2816fe719a4026ffa1ee0189dc89bdfdbafb164.pid for a running server ``` With this in place you then need to trigger vscode to attempt to reconnect again and it should work fine. (VS code will find the pid file and first filter the ps output by the contents of that file which in this case will be no lines. It will then go through the same code path that is missing quoting but ALL_PROCS will now be empty and so not a problem). If you want to check you are hitting this exact issue - and help identify what argument it is that is causing the hang you could try: Create vstest.sh with the following contents: ``` ALL_PROCS=`ps ax` echo $ALL_PROC ``` Then run - note: this might hang in a way that it is unkillable so be prepared for that: ``` strace bash vstest.sh ``` If you are hitting the same issue this should hang and the last strace line should tell you what is unresponsive - perhaps you can then kill it so that it is no longer in the ps output. If it does hang try killing the bash process (with both SIGTERM and SIGKILL if SIGTERM doesn't work) - if SIGKILL doesn't work which can happen if the process goes to D state then either kill the terminal session or background the task - either way that process is now going to hang around until reboot :/ HTH,

…

-Chris

On Fri, 17 Feb 2023 at 11:14, Fabian wrote: Hi, whats the advice? I suddenly started to have this issue on one of my clusters. Nonetheless, it does not happen on my other clusters, only on one of them... I read many issues but nothing has solved the "checking for a running server" problem. The issue started after some automatic update on the vs-server 2-3 days ago..

[eleanorjboyd]

Hello! I apologies for the delay. A few questions:

1. Do you have a smaller reproducable setup? We tried to see the wildcard expansion issue on our machines and did not see it
2. What shell are you on?

You seem to be very knowledgable in this area so any additional information is always helpful. Thanks!

[chuckds]

1. If you've tried the noop.sh "/*/*/*/*/*/*/*" approach and you still don't see the issue then I don't have anymore great ideas. Perhaps you could try creating a hung filesystem e.g. https://stackoverflow.com/questions/58631215/how-to-simulate-a-hanging-filesystem ?
2. bash

While I can understand it would be great to be able to reproduce the issue, do you really need to in this case?
Passing arguments like this in shell scripts isn't good e.g. if you put the problem code into www.shellcheck.net (a useful tool for validating shell scripts) it throws up the SC2086 warning:
SC2086 (info): Double quote to prevent globbing and word splitting

So I think it would be reasonable to fix even it the precise symptoms that I've observed cannot be reproduced.

[eleanorjboyd]

that is a great tool! Thank you for sharing, I will be using that in the future. This is a small and reasonable fix so I am good to mark it as a bug that we will get to. Thanks