[Remote-SSH Bug]: Remote SSH fails at "Waiting for server log..." #8063
Comments
|
Hello! I few questions/ comments
Usually when it doesn't work to connect locally with the same connection string we are using it is due to an issue with your setup and not with the extension (since it doesn't work from the command line). This being said I might have some advice on maybe a setting to change. If you could try removing different args from the connection command that might help you narrow down which command line arg is causing the problem. So if the string is "ssh localhost -D -t" trying it without the -D and -T respectively to see if one of those is the issue. |
|
Thank you for the response.
|
|
By the whole logs do you mean the output tab and then selecting "remote-ssh" from the dropdown? We print out more to our logs that what it seems like you have (like there should be other print out before |
|
Ah yes sorry I wasn't including the full log -- I just edited it to include the part before. |
|
Hi! So it looks like you are back a few versions on the remote-ssh extension- does upgrading do anything? And the log string I am asking for is not in that version of the extension but they are still listed so if you could run this and see the option. ssh -v -T -D 56207 -o ConnectTimeout=30 sh03-ln06.stanford.edu so following these steps using that command string:
|
|
I just installed the latest version of VSCode and the newest Remote-SSH. The same error happens, I've updated the log above. Thanks for including the ssh command -- as written, the connection hangs, but when I drop the |
|
Ok great thank you for finding this. This is a known bug that some people are running into- you can see it here as well: #7673. I have created an issue to fix it here: #7558. If you could go and upvote that issue it would be helpful for keeping track of how many people are affected and prioritization. Also, could you outline any guesses you have about what might have changed between it working and not? We have always included the |
|
Got it thanks! Upvoted and I'll be following the issue. It's weird, I don't have a good sense of what may be causing the issue. I want to say I recently updated VS Code but now when I uninstall and use older versions that used to work I have the same issue. I also regularly use VS Code to log into two clusters but the issue only occurs for one of the clusters, and the other cluster works fine. Moreover I asked one other person who uses the problematic cluster to see if they were also having this issue and they reported the same problem. |
|
Also looking at #7673 I should specify that I'm not running OS Ventura (I'm still on Monterey) but I'm running into the issue. |
|
Thank you for the feedback! Do you know any differences between the two clusters? |
|
I don't have much insight into how they differ under the hood. But one weird thing is that the behavior with and without using the And even when using the With the Without the
|
|
Same problem here, and I'm on the same cluster as @keyonvafa. I think the root cause is not I saw the same seg fault message in my vscode ssh log. After some digging, I found the problem very intriguing.. The I'm clueless at this point... |
|
I once got a segfault launching the server on one of the log in node but not on the compute node. I mostly launched the server on compute node. I was thinking running the server may be taking too much memory on the log in node but @cffan's test show it may not be a memory issue. It is more likely a bug exposed only when the program name becomes long. |
@zyzhang1992 How do you run vscode in compute node? Does that requires some additional ssh tunneling setup? |
|
@eleanorjboyd I think I can confirm that the problem is specific to the log in node. We access the cluster through the log in node and then allocate resources on the compute nodes. The log in nodes use cgroup controls of the resources. There may also be limitations to the stack/heap and file meta data on the log in nodes. Look like that there might be a bug of remote-ssh that are only exposed when the server is launched on the log in nodes. Let us know how we can help resolving this issue. Thanks! |
|
@zyzhang1992 @keyonvafa just FYI, my current work around is to use remote tunnel on the compute node. |
|
@cffan how did you set it up? Do you log on to a compute node and then start the server on the compute node? That is something I was hoping to do but haven't had the time to try that yet. |
Yes. I use |
|
Host sherlock Host sherlock-compute @cffan here is the config I use for lauching the server on the compute node. I have a slurm job with the job name vscode-tunnel for the allocation. I can go over it with you during the OH if you think that may help. |
|
Thank you for the additional information and context. I am going to close this issue now as it seems we have reached an endpoint in this discussion but please reopen if that is not the case. Thanks |
|
I have been having this issue for months now also on the same cluster as @cffan and @keyonvafa and it's also an issue with the login node, and I would very much like a solution. I like having the possibility of connecting to the login node and then connect to interactive sessions from there, so if I can avoid connecting directly to the compute nodes, that would be great. I have a log file that also points to a segfault, but I initially thought if it could be caused by a libstdc++ version mismatch. |
|
that is unlikely since it is the same libraries on both the log in and the compute nodes. segfault only occurs on the log in node, not the compute node.
…________________________________
From: Alexander Neergaard Zahid ***@***.***>
Sent: Wednesday, March 8, 2023 12:36 PM
To: microsoft/vscode-remote-release ***@***.***>
Cc: Zhiyong Zhang ***@***.***>; Mention ***@***.***>
Subject: Re: [microsoft/vscode-remote-release] [Remote-SSH Bug]: Remote SSH fails at "Waiting for server log..." (Issue #8063)
I have been having this issue for months now also on the same cluster as @cffan<https://github.com/cffan> and @keyonvafa<https://github.com/keyonvafa> and it's also an issue with the login node.
I have a log file that also points to a segfault, but I was wondering if it could be caused by a libstdc++ version mismatch?
vscode_remote_log.txt<https://github.com/microsoft/vscode-remote-release/files/10924987/vscode_remote_log.txt>
See here<https://code.visualstudio.com/docs/remote/linux#_remote-host-container-wsl-linux-prerequisites>
—
Reply to this email directly, view it on GitHub<#8063 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADG474FIJIUBEGJOSOTAS6DW3DUUXANCNFSM6AAAAAAVAQP7W4>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Is there an existing issue for this bug?
Required Troubleshooting Steps
remote.SSH.useLocalServersettingConnect Locally
It doesn’t connect successfully (same error message)
->
No response
Expected Behavior
I've been using Remote for the past two years to SSH in to another server without problems. However in the last week I am suddenly unable to connect. I've tried the usual suspects to no avail:
~/.vscode_serverMoreover, this issue doesn't occur when I ssh to the server via the command line -- it only happens when I connect with Remote.
Steps To Reproduce
When I try connecting to Remote, the connection hangs -- I enter my password and provide 2 factor authentication successfully, but it gets stuck at "Waiting for server log..."
Remote-SSH Log
Remote-SSH Log
Anything else?
Thank you!
The text was updated successfully, but these errors were encountered: