Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connecting with SSH timed out #1031

Closed
raine-works opened this issue Apr 26, 2024 · 22 comments
Closed

Connecting with SSH timed out #1031

raine-works opened this issue Apr 26, 2024 · 22 comments

Comments

@raine-works
Copy link

raine-works commented Apr 26, 2024

What happened?
Attempting to connect to provider remote host via SSH times out with error: "Waiting for Devpod agent to come up..." Cannot connect via SSH with VSCode or the terminal. The Devpod CLI SSH command works as intended.

What did you expect to happen instead?
Devcontainer to be accessible via SSH.

How can we reproduce the bug? (as minimally and precisely as possible)

My devcontainer.json:

{
	"name": "Homelab",
	"build": {
		"dockerfile": "Dockerfile",
		"context": ".",
		"args": {
			"USERNAME": "${localEnv:USER}"
		}
	},
	"remoteUser": "${localEnv:USER}",
	"features": {
		"ghcr.io/devcontainers/features/github-cli:1": {}
	},
	"forwardPorts": [3000],
	"otherPortsAttributes": {
		"onAutoForward": "ignore"
	},
	"mounts": [
		"source=${env:HOME}${env:USERPROFILE}/.config/gh,target=/home/${localEnv:USER}/.config/gh,type=bind",
		"source=${env:HOME}${env:USERPROFILE}/.ssh,target=/home/${localEnv:USER}/.ssh,type=bind"
	],
	"postAttachCommand": "git config --global diff.tool ...",
	"postStartCommand": "pnpm config set store-dir ~/.pnpm-store && pnpm setup",
	"customizations": {
		"vscode": {
			"extensions": [
				"esbenp.prettier-vscode",
				"yoavbls.pretty-ts-errors",
				"dbaeumer.vscode-eslint",
				"bradlc.vscode-tailwindcss",
				"eamodio.gitlens",
				"ms-azuretools.vscode-docker"
			]
		}
	},
	"runArgs": ["--network=host"]
}

My Dockerfile:

FROM mcr.microsoft.com/devcontainers/typescript-node:20 AS base

ARG USERNAME
ENV SHELL="/bin/bash"

RUN apt update
RUN apt upgrade -y
RUN usermod -l ${USERNAME} -d /home/${USERNAME} -m node
RUN echo ${USERNAME} ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/${USERNAME}

My SSH config:

# DevPod Start homelab.devpod
Host homelab.devpod
  ForwardAgent yes
  LogLevel error
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
  ProxyCommand "C:\Program Files\DevPod\devpod-cli.exe" ssh --stdio --context default --user raine homelab
  User raine
# DevPod End homelab.devpod

Host virgo.lab.raineworks.com
  HostName virgo.lab.raineworks.com
  User raine

Local Environment:

  • devpod 0.5.5
  • Operating System: windows
  • ARCH of the OS: AMD64

DevPod Provider:

  • Remote provider: ssh

Anything else we need to know?
I am able to connect to the remote host via SSH directly.

Docker engine on remote host has the following custom remote access config:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://virgo.lab.raineworks.com:2375
@pascalbreuninger
Copy link
Member

@raine-works are you experiencing the same problem with the latest v0.5.6 release?

@raine-works
Copy link
Author

raine-works commented Apr 29, 2024

I updated to v0.5.6 and I am still seeing the same error.

PS C:\Users\Raine> ssh homelab.devpod
13:40:39 info Waiting for devpod agent to come up...
13:41:02 info Waiting for devpod agent to come up...

I tried deleting that workspace and creating a new one after updating. I am seeing the same results. The devpod cli ssh command works but the standard ssh command does not.

@89luca89
Copy link
Contributor

Hi @raine-works can you please to a test for me?

Can you open the ssh config file (It should be $HOME\.ssh\config also on windows) and add a --debug to the ProxyCommand of devpod?

Like:

...
...
  ProxyCommand devpod ssh --debug --stdio --context default --user vscode distrobox
  User vscode

So we can get some more useful logs from your command?

@raine-works
Copy link
Author

raine-works commented Apr 30, 2024

Here is my SSH config.

# DevPod Start hukt.devpod
Host hukt.devpod
  ForwardAgent yes
  LogLevel error
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
  ProxyCommand "C:\Program Files\DevPod\devpod-cli.exe" ssh --debug --stdio --context default --user raine hukt
  User raine
# DevPod End hukt.devpod

Host virgo.lab.raineworks.com
  HostName virgo.lab.raineworks.com
  User raine

Here are the logs I am seeing after adding the --debug flag.

PS C:\Users\Raine> ssh hukt.devpod
12:21:05 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
12:21:05 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
12:21:05 debug Error adding key C:\Users\Raine\.ssh\id_rsa to agent: Error connecting to agent: No such file or directory

exit status 2
12:21:05 debug Acquire workspace lock...
12:21:05 debug Acquired workspace lock...
12:21:05 debug execute inject script
12:21:05 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
12:21:05 debug Run command provider command: ${SSH_PROVIDER} command
12:21:25 debug done inject
12:21:25 debug done injecting
12:21:25 info Waiting for devpod agent to come up...

The "inject script" portion repeats endlessly.

@89luca89
Copy link
Contributor

89luca89 commented May 2, 2024

Thanks @raine-works
can you update the ssh provider to the latest version? we implemented some fixes for how it behaves and those could potentially help for this issue

@raine-works
Copy link
Author

raine-works commented May 2, 2024

I assume this can be done by deleting the provider and recreating it? I did that and Its still not working. I am seeing the same error message.

@89luca89
Copy link
Contributor

89luca89 commented May 2, 2024

Can you post it?

@raine-works
Copy link
Author

PS C:\Users\Raine> ssh hukt.devpod
11:53:18 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
11:53:18 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
11:53:18 debug Error adding key C:\Users\Raine\.ssh\id_rsa to agent: Error connecting to agent: No such file or directory

exit status 2
11:53:18 debug Acquire workspace lock...
11:53:18 debug Acquired workspace lock...
11:53:18 debug execute inject script
11:53:18 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
11:53:18 debug Run command provider command: ${SSH_PROVIDER} command
11:53:38 debug done inject
11:53:38 debug done injecting
11:53:38 info Waiting for devpod agent to come up...
11:53:38 debug Inject Error: context deadline exceeded
11:53:38 debug done exec
11:53:41 debug execute inject script
11:53:41 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
11:53:41 debug Run command provider command: ${SSH_PROVIDER} command
Host hukt.devpod
  ForwardAgent yes
  LogLevel error
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
  ProxyCommand "C:\Program Files\DevPod\devpod-cli.exe" ssh --debug --stdio --context default --user raine hukt
  User raine
# DevPod End hukt.devpod

Host virgo.lab.raineworks.com
  HostName virgo.lab.raineworks.com
  User raine

@shanman190
Copy link
Contributor

shanman190 commented May 6, 2024

@raine-works, what OpenSSH version are you using? ssh -V

The default OpenSSH version from Windows is VERY old (8.1), so downloading the latest release from PowerShell/Win32-OpenSSH is probably necessary. Alternatively, if you don't make use of the SSH agent, you can disable the Windows service to fix it as well.

@raine-works
Copy link
Author

OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3

@shanman190
Copy link
Contributor

Would you be open to upgrading by following this guide from Microsoft?
https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH

It's most likely that the source (8.6) is incompatible with the target (likely 9.x).

@raine-works
Copy link
Author

raine-works commented May 7, 2024

I updated to OpenSSH version 9.5.

PS C:\Users\Raine> ssh -V
OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2

I am still seeing a similar error.

PS C:\Users\Raine> ssh hukt.devpod
10:49:24 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
10:49:24 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
10:49:24 debug Acquire workspace lock...
10:49:24 debug Acquired workspace lock...
10:49:24 debug execute inject script
10:49:24 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:49:24 debug Run command provider command: ${SSH_PROVIDER} command
10:49:44 debug done inject
10:49:44 debug done injecting
10:49:44 info Waiting for devpod agent to come up...
10:49:44 debug Inject Error: context deadline exceeded
10:49:44 debug done exec
10:49:47 debug execute inject script
10:49:47 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:49:47 debug Run command provider command: ${SSH_PROVIDER} command

@shanman190
Copy link
Contributor

So that "error" might happen a couple of times while the agent process is getting started. Then it normally goes away once the agent is ready to accept commands. Does it just continue to spin like that indefinitely?

@raine-works
Copy link
Author

raine-works commented May 7, 2024

The longest I've let it run is for about 5 minutes. This is the log output after 3 minutes.

PS C:\Users\Raine> ssh hukt.devpod
10:57:09 debug Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
10:57:09 debug Run ssh-add C:\Users\Raine\.ssh\id_rsa
10:57:09 debug Acquire workspace lock...
10:57:09 debug Acquired workspace lock...
10:57:09 debug execute inject script
10:57:09 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:57:09 debug Run command provider command: ${SSH_PROVIDER} command
10:57:29 debug done inject
10:57:29 debug done injecting
10:57:29 info Waiting for devpod agent to come up...
10:57:29 debug Inject Error: context deadline exceeded
10:57:29 debug done exec
10:57:32 debug execute inject script
10:57:32 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:57:32 debug Run command provider command: ${SSH_PROVIDER} command
10:57:52 debug done inject
10:57:52 debug done injecting
10:57:52 info Waiting for devpod agent to come up...
10:57:52 debug Inject Error: context deadline exceeded
10:57:52 debug done exec
10:57:55 debug execute inject script
10:57:55 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:57:55 debug Run command provider command: ${SSH_PROVIDER} command
10:58:15 debug done inject
10:58:15 debug done injecting
10:58:15 info Waiting for devpod agent to come up...
10:58:15 debug Inject Error: context deadline exceeded
10:58:15 debug done exec
10:58:18 debug execute inject script
10:58:18 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:58:18 debug Run command provider command: ${SSH_PROVIDER} command
10:58:38 debug done inject
10:58:38 debug done injecting
10:58:38 info Waiting for devpod agent to come up...
10:58:38 debug Inject Error: context deadline exceeded
10:58:38 debug done exec
10:58:41 debug execute inject script
10:58:41 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:58:41 debug Run command provider command: ${SSH_PROVIDER} command
10:59:01 debug done inject
10:59:01 debug done injecting
10:59:01 info Waiting for devpod agent to come up...
10:59:01 debug Inject Error: context deadline exceeded
10:59:01 debug done exec
10:59:04 debug execute inject script
10:59:04 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:59:04 debug Run command provider command: ${SSH_PROVIDER} command
10:59:24 debug done inject
10:59:24 debug done injecting
10:59:24 info Waiting for devpod agent to come up...
10:59:24 debug Inject Error: context deadline exceeded
10:59:24 debug done exec
10:59:27 debug execute inject script
10:59:27 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:59:27 debug Run command provider command: ${SSH_PROVIDER} command
10:59:47 debug done inject
10:59:47 debug done injecting
10:59:47 info Waiting for devpod agent to come up...
10:59:47 debug Inject Error: context deadline exceeded
10:59:47 debug done exec
10:59:50 debug execute inject script
10:59:50 debug download agent from https://github.com/loft-sh/devpod/releases/download/v0.5.6
10:59:50 debug Run command provider command: ${SSH_PROVIDER} command
11:00:10 debug done inject
11:00:10 debug done injecting
11:00:10 info Waiting for devpod agent to come up...
11:00:10 debug Inject Error: context deadline exceeded
11:00:10 debug done exec

@Sword-Breaker
Copy link

I think there is a typo in 'https://github.com/loft-sh/devpod/releases/download/v0.5.6', the URL is 404.

@raine-works
Copy link
Author

raine-works commented May 9, 2024

The default agent URL under general in the settings is https://github.com/loft-sh/devpod/releases/latest/download/ which redirects to https://github.com/loft-sh/devpod/releases/download/v0.5.6. What should the override agent URL be?

@shanman190
Copy link
Contributor

So that download URL is just the first fragment. The file name gets appended to it based upon the target systems OS and architecture.

It was a little misleading when I was debugging that area as well, but that part is just fine from what I've seen.

@raine-works
Copy link
Author

raine-works commented May 12, 2024

I removed the agent from the remote machine and tried re-creating the workspace. The agent was successfully re-installed on the remote machine, but still hangs when trying to open a connection.

@raine-works
Copy link
Author

I find it strange that the devpod cli ssh command can connect to the remote devcontainer but open-ssh in powershell can't.

PowerShell 7.4.2

Raine  ~  ♥ 16:04  ssh node-test.devpod
16:04:29 info Waiting for devpod agent to come up...
16:04:52 info Waiting for devpod agent to come up...
16:05:15 info Waiting for devpod agent to come up...
16:05:38 info Waiting for devpod agent to come up...
16:06:01 info Waiting for devpod agent to come up...
16:06:24 info Waiting for devpod agent to come up...
16:06:47 info Waiting for devpod agent to come up...
16:07:10 info Waiting for devpod agent to come up...
16:07:33 info Waiting for devpod agent to come up...
16:07:56 info Waiting for devpod agent to come up...
16:08:19 info Waiting for devpod agent to come up...
16:08:42 info Waiting for devpod agent to come up...
16:09:05 info Waiting for devpod agent to come up...
16:09:28 fatal connect to server: timeout waiting for instance connection: context deadline exceeded

Raine  ~  ♥ 16:09  devpod node-test
16:10:35 error Try using the --debug flag to see a more verbose output
16:10:35 fatal unknown command "node-test" for "devpod"

Raine  ~  ♥ 16:10  devpod ssh node-test
node ➜ ~ $ ls
node ➜ ~ $ ls -a
.   .bash_logout  .cache   .docker     .npm        .profile        .zprofile
..  .bashrc       .config  .gitconfig  .oh-my-zsh  .vscode-server  .zshrc
node ➜ ~ $ cd /workspaces/
node ➜ /workspaces $ ls
node-test
node ➜ /workspaces $ cd node-test/
node ➜ /workspaces/node-test (main) $ ls
CODE_OF_CONDUCT.md  LICENSE  README.md  SECURITY.md  node_modules  package.json  server.js  yarn.lock
node ➜ /workspaces/node-test (main) $ npm run start

> docker_web_app@1.0.0 start
> node server.js

Running on http://0.0.0.0:3000

@raine-works
Copy link
Author

raine-works commented May 16, 2024

@shanman190

I copied the SSH config from the Windows file system and pasted it in the SSH config in WSL. I adjusted the proxy command to use /mnt/c.

# DevPod Start node-test.devpod
Host node-test.devpod
  ForwardAgent yes
  LogLevel error
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
  ProxyCommand "/mnt/c/Program Files/DevPod/devpod-cli.exe" ssh --stdio --context default --user node node-test
  User node
# DevPod End node-test.devpod

In WSL it worked immediately.

OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2
OpenSSH_8.9p1 Ubuntu-3ubuntu0.7, OpenSSL 3.0.2 15 Mar 2022

My issue must be related to the open-ssh version like you suggested before. Although updating to 9.x didn't solve the problem.

@thomasxd24
Copy link

@raine-works try using the new version of the ssh provider, I had the same problem as you. The new version fixed the problem for me.

(Even though it introduced another bug :/)

@raine-works
Copy link
Author

Hallelujah! It works!!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants