[Remote-SSH Bug]: Hangs at "Checking for a running server"

[whugimy]

Is there an existing issue for this bug?

• I have searched the existing issues

Required Troubleshooting Steps

• I have followed these troubleshooting steps
• I have tried both values of the remote.SSH.useLocalServer setting

Connect Locally

It connects successfully

->

No response

Expected Behavior

Expected to successfully login to my account on a remote system via ssh and access its files.

Steps To Reproduce

Sorry, I don't have too good of an idea of where to find details, but I will provide as much as possible and do my best to provide more info if needed.

1. On a Macbook running MacOS 14.3.1
2. Using VScode Version 1.86.1
3. Using Remote-SSH v0.108.0
4. Connecting to a Linux server with the following info:

Linux Version Info

NAME="Fedora Linux"
VERSION="38 (Thirty Eight)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora Linux 38 (Thirty Eight)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f38/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14

5. VSCode on the Linux server is 1.86.0

When trying to login through VSC's Remote-SSH to one specific remote server, the program hangs on a line where the system is trying to check a .log and a .pid file within the server directory.

I am able to connect to the system via my Macbook's native terminal, and it seems like those files that Remote-SSH is trying to access don't exist. The specific files are shown below in the log.

I have tried the following steps:

1. Removing .vscode-server/ and having Remote-SSH reinstall it
2. Uninstalling and reinstalling Remote-SSH
3. Repeating the above steps with older versions of Remote-SSH
4. Uninstalling and reinstalling VSCode in its entirety
5. Installing older versions of VSCode
6. Copying .vscode-server/ from a server that I am able to connect to to the one that I cannot connect to
7. Installing .vscode-server/ directly from https://update.code.visualstudio.com/commit:31c37ee8f63491495ac49e43b8544550fbae4533/server-linux-x64/stable and unzipping into the remote device, then reconnecting with Remote-SSH
8. Even reset Macbook to factory settings and reinstalled VSCode again

None of these steps have worked. They have consistently produced the same result on my computer: I am unable to login because the login process hangs when looking for a running server.

I am not able to reproduce this issue on other computers OR even on other remote systems from this computer.

I actually attempted to connect to the system using other computers' VSCode installations: they all seem to work fine. However, even after a friend's laptop is successfully able to connect and create the missing files, when attempting to connect again from my laptop, the connection process hangs with the following line:

> Looking for server with pid: 2860232

I assume this pid does not function properly with my computer.

And again, I am also able to successfully connect to other systems that I have a user account on—they do not have the same issue, only this specific device.

I'm not sure where to look to find more info, but I am happy to provide as much as I can, as using VSCode on this device to remotely access this specific machine is extremely important to me.

Remote-SSH Log

Remote-SSH Log of Failed Attempt From My Computer

[23:50:38.250] Log Level: 2
[23:50:38.253] SSH Resolver called for "ssh-remote+csil.cs.ucsb.edu", attempt 1
[23:50:38.253] "remote.SSH.useLocalServer": false
[23:50:38.253] "remote.SSH.useExecServer": false
[23:50:38.254] "remote.SSH.showLoginTerminal": true
[23:50:38.254] "remote.SSH.remotePlatform": {}
[23:50:38.254] "remote.SSH.path": undefined
[23:50:38.254] "remote.SSH.configFile": undefined
[23:50:38.254] "remote.SSH.useFlock": true
[23:50:38.254] "remote.SSH.lockfilesInTmp": false
[23:50:38.254] "remote.SSH.localServerDownload": auto
[23:50:38.254] "remote.SSH.remoteServerListenOnSocket": false
[23:50:38.254] "remote.SSH.showLoginTerminal": true
[23:50:38.254] "remote.SSH.defaultExtensions": []
[23:50:38.254] "remote.SSH.loglevel": 2
[23:50:38.254] "remote.SSH.enableDynamicForwarding": true
[23:50:38.254] "remote.SSH.enableRemoteCommand": false
[23:50:38.254] "remote.SSH.serverPickPortsFromRange": {}
[23:50:38.254] "remote.SSH.serverInstallPath": {}
[23:50:38.255] VS Code version: 1.86.1
[23:50:38.255] Remote-SSH version: remote-ssh@0.108.0
[23:50:38.255] darwin arm64
[23:50:38.256] SSH Resolver called for host: csil.cs.ucsb.edu
[23:50:38.256] Setting up SSH remote "csil.cs.ucsb.edu"
[23:50:38.258] Using commit id "31c37ee8f63491495ac49e43b8544550fbae4533" and quality "stable" for server
[23:50:38.260] Install and start server if needed
[23:50:40.211] PATH: /usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
[23:50:40.211] Checking ssh with "ssh -V"
[23:50:40.220] > OpenSSH_9.4p1, LibreSSL 3.3.6

[23:50:40.222] Running script with connection command: ssh -T -D 49570 -o ConnectTimeout=15 'csil.cs.ucsb.edu' bash
[23:50:40.519] > csilvm-02.cs.ucsb.edu
> michaelyang@csil.cs.ucsb.edu's password: 
[23:50:40.519] Got some output, clearing connection timeout
[23:50:45.141] > 
[23:50:46.450] > 4e0a33ca0d85: running
[23:50:46.512] > Acquiring lock on /cs/student/michaelyang/.vscode-server/bin/31c37ee8f63491495ac49e43b8544550fbae4533/vscode-remote-lock.michaelyang.31c37ee8f63491495ac49e43b8544550fbae4533
[23:50:46.528] > Found existing installation at /cs/student/michaelyang/.vscode-server/bin/31c37ee8f63491495ac49e43b8544550fbae4533...
> Checking /cs/student/michaelyang/.vscode-server/.31c37ee8f63491495ac49e43b8544550fbae4533.log and /cs/student/michaelyang/.vscode-server/.31c37ee8f63491495ac49e43b8544550fbae4533.pid for a running server

Remote-SSH Log of Failed Attempt From My Computer After Successful Access From Other Computer

[00:04:17.255] Log Level: 2
[00:04:17.259] SSH Resolver called for "ssh-remote+csil.cs.ucsb.edu", attempt 1
[00:04:17.259] "remote.SSH.useLocalServer": false
[00:04:17.259] "remote.SSH.useExecServer": false
[00:04:17.259] "remote.SSH.showLoginTerminal": true
[00:04:17.259] "remote.SSH.remotePlatform": {}
[00:04:17.259] "remote.SSH.path": undefined
[00:04:17.260] "remote.SSH.configFile": undefined
[00:04:17.260] "remote.SSH.useFlock": true
[00:04:17.260] "remote.SSH.lockfilesInTmp": false
[00:04:17.260] "remote.SSH.localServerDownload": auto
[00:04:17.260] "remote.SSH.remoteServerListenOnSocket": false
[00:04:17.260] "remote.SSH.showLoginTerminal": true
[00:04:17.260] "remote.SSH.defaultExtensions": []
[00:04:17.260] "remote.SSH.loglevel": 2
[00:04:17.260] "remote.SSH.enableDynamicForwarding": true
[00:04:17.260] "remote.SSH.enableRemoteCommand": false
[00:04:17.260] "remote.SSH.serverPickPortsFromRange": {}
[00:04:17.260] "remote.SSH.serverInstallPath": {}
[00:04:17.261] VS Code version: 1.86.1
[00:04:17.261] Remote-SSH version: remote-ssh@0.108.0
[00:04:17.261] darwin arm64
[00:04:17.262] SSH Resolver called for host: csil.cs.ucsb.edu
[00:04:17.262] Setting up SSH remote "csil.cs.ucsb.edu"
[00:04:17.263] Using commit id "31c37ee8f63491495ac49e43b8544550fbae4533" and quality "stable" for server
[00:04:17.266] Install and start server if needed
[00:04:18.221] PATH: /usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
[00:04:18.221] Checking ssh with "ssh -V"
[00:04:18.230] > OpenSSH_9.4p1, LibreSSL 3.3.6

[00:04:18.232] Running script with connection command: ssh -T -D 49654 -o ConnectTimeout=15 'csil.cs.ucsb.edu' bash
[00:04:18.498] > csilvm-02.cs.ucsb.edu
> michaelyang@csil.cs.ucsb.edu's password: 
[00:04:18.498] Got some output, clearing connection timeout
[00:04:21.181] > 
[00:04:22.246] > b68d94b5d015: running
[00:04:22.284] > Acquiring lock on /cs/student/michaelyang/.vscode-server/bin/31c37ee8f63491495ac49e43b8544550fbae4533/vscode-remote-lock.michaelyang.31c37ee8f63491495ac49e43b8544550fbae4533
[00:04:22.294] > Found existing installation at /cs/student/michaelyang/.vscode-server/bin/31c37ee8f63491495ac49e43b8544550fbae4533...
> Checking /cs/student/michaelyang/.vscode-server/.31c37ee8f63491495ac49e43b8544550fbae4533.log and /cs/student/michaelyang/.vscode-server/.31c37ee8f63491495ac49e43b8544550fbae4533.pid for a running server
> Looking for server with pid: 2860232

Although logs do not reflect it, I have attempted both values of remote.SSH.useLocalServer

Anything else?

Sorry, I am a student and not super experienced with where I might find more necessary info on my own system! Am happy to respond and provide as much info as I can

[roblourens]

Not sure why it's hanging there. Can you try connecting with the setting "remote.SSH.useExecServer": true?

The next line would only be ps ax with a series of grep, tr, wc to find a certain process, anything special about those commands on your remote host?

[bradenkwebb]

I'm experiencing the same issue, running a local setup of Windows 10 and a remote CentOS Linux 7. I used the same remote-ssh process to access this machine several times a week for the last several months without problems until the VS Code update to version 1.86. Since I believe that version is now incompatible with CentOS 7 remotes, I had to revert my local install to version 1.85.2. This worked initially, but the next day that I tried to log in, I experienced a couple variants of this problem. I've tried toggling on and off a few different Remote-SSH settings, including the remote.SSH.useExecServer and remote.SSH.useLocalServer.

Local VS Code info:

Version: 1.85.2 (user setup)
Commit: 8b3775030ed1a69b13e4f4c628c612102e30a681
Date: 2024-01-18T06:40:10.514Z
Electron: 25.9.7
ElectronBuildId: 26354273
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: 11.4.183.29-electron.0
OS: Windows_NT x64 10.0.19045

With remote.SSH.useExecServer=True, remote.SSH.useFlock=True, and remote.SSH.useLocalServer as either True or False, I get the following logs:

tail end of logs:

[12:43:36.866] Remote server is listening on port 39146
[12:43:36.866] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":39146},"osReleaseId":"centos","arch":"x86_64","sshAuthSock":"","display":"","tmpDir":"/run/user/315993","platform":"linux","execServerToken":"217f2dbda40d"},"downloadTime":429,"installTime":346,"serverStartTime":38,"installUnpackCode":"success"}
[12:43:36.869] Starting forwarding server. local port 62696 -> socksPort 62692 -> remotePort 39146
[12:43:36.871] Forwarding server listening on port 62696
[12:43:36.872] Waiting for ssh tunnel to be ready
[12:43:36.875] [Forwarding server port 62696] Got connection 0
[12:43:36.877] Tunneled port 39146 to local port 62696
[12:43:36.878] Resolved "ssh-remote+deception" to "port 62696"
[12:43:36.890] Initizing new exec server for ssh-remote+deception
[12:43:36.890] Resolving exec server at port 62696
[12:43:36.916] [Forwarding server port 62696] Got connection 1
[12:43:37.192] Exec server for ssh-remote+deception created and cached
[12:43:37.209] ------

[12:43:37.319] [server] Checking /<homedir>/.vscode-server/cli/servers/Stable-8b3775030ed1a69b13e4f4c628c612102e30a681/log.txt and /<homedir>/.vscode-server/cli/servers/Stable-8b3775030ed1a69b13e4f4c628c612102e30a681/pid.txt for a running server...

[whugimy]

Not sure why it's hanging there. Can you try connecting with the setting "remote.SSH.useExecServer": true?

The next line would only be ps ax with a series of grep, tr, wc to find a certain process, anything special about those commands on your remote host?

Set the value to true and the same issue occurs; it is still checking nonexistant files for a running server. This hangs until I ssh into the machine from my terminal and manually kill all vscode-related processes with pkill -9

[10:31:16.435] [server] Checking /cs/student/michaelyang/.vscode-server/cli/servers/Stable-31c37ee8f63491495ac49e43b8544550fbae4533/log.txt and /cs/student/michaelyang/.vscode-server/cli/servers/Stable-31c37ee8f63491495ac49e43b8544550fbae4533/pid.txt for a running server...

ps ax seems to be hanging here:

openat(AT_FDCWD, "/proc/262609/cmdline", O_RDONLY) = 6
read(6, 

Will see if I can get a hold of my sysadmin to ask about my account as well.

[bradenkwebb]

@whugimy I ran into this issue yesterday, changed a few settings, and somehow was able to bypass it and connect successfully. So while this definitely seems like a bug, there should be a workaround! But I've had no luck reproducing that connection today.

[whugimy]

@whugimy I ran into this issue yesterday, changed a few settings, and somehow was able to bypass it and connect successfully. So while this definitely seems like a bug, there should be a workaround! But I've had no luck reproducing that connection today.

Hi @bradenkwebb, Do you mind me asking what some of those settings you had to change were? I'm not having any luck.

[bradenkwebb]

The settings that I know I've touched include

• remote.SSH.localServerDownload
• remote.SSH.useExecServer
• remote.SSH.useFlock
• remote.SSH.useLocalServer

as well as either deleting the entire .vscode-server directory on the remote machine or hitting >Remote-SSH: Kill VS Code Server on Host in between tries. None of that seems to be making much of a difference right now, though

[bradenkwebb]

@whugimy I finally got it working! I think the key was to use the command

>Remote-SSH: Uninstall VS Code Server from Host

before connecting, instead of just killing or deleting the pre-existing server directory. Not sure what the underlying difference is, but I'm connected.

The only other changes I currently have in my settings.json are these (I was experimenting with the lockfiles settings, but I'm not sure if that was the fix or not):

{
    "remote.SSH.remotePlatform": {
            "<hostname>": "linux"
     },
    "remote.SSH.useExecServer": true,
    "remote.SSH.lockfilesInTmp": true
}

[whugimy]

@bradenkwebb I've tried running that command and using those settings, but it still doesn't seem to be working for me. Still looking for files that don't exist:

Checking /cs/student/michaelyang/.vscode-server/cli/servers/Stable-31c37ee8f63491495ac49e43b8544550fbae4533/log.txt and /cs/student/michaelyang/.vscode-server/cli/servers/Stable-31c37ee8f63491495ac49e43b8544550fbae4533/pid.txt for a running server...

Still odd, because I just tried connecting from another computer—and it works fine! All the files it's looking for are generated properly.

Another edit: I just tried using the same laptop and Remote-SSH'd to another system—it also worked fine. Very lost now.

[whugimy]

Worked after I used my system's VPN—way above my paygrade, but I spoke to my system admin and it seems like there was probably something related to the firewalls blocking certain portions of the server installs. Using the VPN allowed the server files to install properly, then all connections worked fine.

[hariesramdhani]

Hi I encountered, the same issue and tried all of the work around but the VPN one and still get the same problem

[10:51:44.876] Remote server is listening on port 38177
[10:51:44.876] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":38177},"osReleaseId":"rhel","arch":"x86_64","sshAuthSock":"","display":"","tmpDir":"/run/user/193544","platform":"linux","execServerToken":"aaa1111a-a111-11a1-a111-111aa1111111"},"serverStartTime":146,"installUnpackCode":""}
[10:51:44.876] Persisting server connection details to $LOCAL_HOME/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-0b74131d-863d2581ecda6849923a2118d93a088b0745d9d6-0.109.0-es/data.json
[10:51:44.879] Starting forwarding server. local port 50839 -> socksPort 50836 -> remotePort 38177
[10:51:44.879] Forwarding server listening on port 50839
[10:51:44.879] Waiting for ssh tunnel to be ready
[10:51:44.880] Tunneled port 38177 to local port 50839
[10:51:44.880] Resolved "ssh-remote+maxlogin1" to "port 50839"
[10:51:44.887] Initizing new exec server for ssh-remote+maxlogin1
[10:51:44.887] Resolving exec server at port 50839
[10:51:44.902] [Forwarding server port 50839] Got connection 0
[10:51:44.903] [Forwarding server port 50839] Got connection 1
[10:51:44.968] Exec server for ssh-remote+maxlogin1 created and cached
[10:51:44.972] ------
[10:51:44.981] [server] Checking $HOME/.vscode-server/cli/servers/Stable-863d2581ecda6849923a2118d93a088b0745d9d6/log.txt and $HOME/.vscode-server/cli/servers/Stable-863d2581ecda6849923a2118d93a088b0745d9d6/pid.txt for a running server...