Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does not support agent forwarding - committing on remote host via SSH fails #16

Open
rcarmo opened this issue May 2, 2019 · 30 comments

Comments

Projects
None yet
@rcarmo
Copy link

commented May 2, 2019

  • VSCode Version: Version 1.34.0-insider (1.34.0-insider)
  • Local OS Version: Mac OS X Mojave 10.14.4 (18E226)
  • Remote OS Version: Ubuntu Linux 18.04
  • Remote Extension/Connection Type: SSH

Steps to Reproduce:

  1. Open remote folder/project workspace with git repository via SSH
  2. Try to commit something to an even more remote repo (say GitHub) that relies on SSH auth.

Since the remote server does not have your SSH key and the VS Code connection does not use agent forwarding, key validation by the Git server (GitHub in this case) fails because the remote machine cannot talk to your local SSH agent and does not have valid keys.

This is a breaking issue for anyone who has to work via jump boxes.

Does this issue occur when you try this locally?: N/A
Does this issue occur when you try this locally and all extensions are disabled?: N/A

@kieferrm

This comment has been minimized.

Copy link
Contributor

commented May 2, 2019

@rcarmo If you ignore VS Code Remote for a bit, how have you setup your SSH box so that your git cli on the SSH box works well over ssh to GitHub?

@rcarmo

This comment has been minimized.

Copy link
Author

commented May 2, 2019

The top of the ~/.ssh/config on my Mac and WSL userland reads like this:

Host *
    AddKeysToAgent yes
    UseKeychain yes
    IdentityFile ~/.ssh/id_rsa
    controlmaster auto
    controlpath /tmp/ssh-%r@%h:%p
    ForwardAgent yes
    Compression yes
    ServerAliveInterval 120
    ServerAliveCountMax 3
    TCPKeepAlive yes

VSCode on the Mac is parsing this file correctly, since it is actually using the hosts I have defined in it as suggestions.

In general, doing what I do (SSHing to a remote machine and then committing via SSH to GitHub using the key on my local laptop) works as long as I have an equivalent local config to the above or explicitly type ssh -A for agent forwarding. This works in Linux (since time imemorial, that specific file above is easily around eight years old), macOS or WSL.

(Just to make it crystal clear, I have to do zero setup on the remote/intermediate box. No extra keys, no added configs, nothing. OpenSSH knows how to deal with agent forwarding and ask my local agent on my laptop for keys, as long as my local machine tells it it's doing agent forwarding)

The only occasion I've had it break is when I forget to add my .tmuxrc to local and remote machines to pass on (and update) the right environment variables when I open a new tmux window.

Besides committing to remote repos, using agent forwarding is also (very) commonly used to log in to machines beyond jump boxes - and the terminal inside VS Code Remote, lacking the right environment variables, cannot talk to my local agent either.

@TomFrost

This comment has been minimized.

Copy link

commented May 3, 2019

Piling on here -- I have agent-forwarding explicitly defined in the host configuration that VSCode is connecting to:

Host work-remote
    User tom
    HostName tom.myworkdomain.net
    IdentityFile ~/.ssh/id_rsa
    ForwardAgent yes
    LocalForward 127.0.0.1:5432 127.0.0.1:5432

VSCode observes all of these things, but ignores ForwardAgent. This is befuddling, since the docs insinuate that VSCode is using the locally installed ssh command, and if I open a terminal window on this machine and ssh work-remote with no other arguments, my agent gets forwarded automatically.

It would be a very high-impact change for me and others I work with to allow the local ssh's default functionality to handle this. Or, alternatively, if VSCode Remote must parse this file and handle the connection itself, passing -A to ssh when ForwardAgent yes is present would also get the job done.

(Quick note, since it's easy to see a bunch of bug reports and feel unappreciated -- this project is amazing and has, in the span of the 4 minutes it took to set up, redefined my development process. This is seriously incredible work, team.)

@roblourens

This comment has been minimized.

Copy link
Member

commented May 3, 2019

This doesn't work by default because what we do isn't quite the same as logging in with a terminal. The ForwardAgent property causes SSH to set the SSH_AUTH_SOCK environment variable in the remote environment. First we open a connection to install the remote agent. If the agent is not already running, we start it. The agent and the remote EH processes that it spawns will get the SSH_AUTH_SOCK from this very first connection. But that SSH connection is short lived so the value of SSH_AUTH_SOCK is invalid later.

But once the agent is running and we know which port it is listening on, we start a second SSH connection to forward that port to the local machine. That SSH connection stays open and we want SSH_AUTH_SOCK from that connection. So we need to

  • Log $SSH_AUTH_SOCK when creating the port tunnel
  • Somehow set that variable in the environment of the remote EH

Remote-SSH is a local extension so we can't do this directly but I think the way to do it would be to have a remote helper extension, send the socket path to it, and have it modify the remote EH environment?

@nathan-sain

This comment has been minimized.

Copy link

commented May 3, 2019

@roblourens I didn't see your answer while I was writing this....

Using VSCode insiders build on Windows 10 using native Windows OpenSSH, I also am unable to use agent forwarding while connected via Remote-SSH. Checking the environment in a VSCode terminal shows

SSH_AUTH_SOCK=/tmp/ssh-m7litxvEqu/agent.56974
SSH_CLIENT='XXX.XXX.XXX.XXX 50429 22'
SSH_CONNECTION='XXX.XXX.XXX.XXX 50429 10.10.1.33 22'

But the path /tmp/ssh-m7litxvEqu/agent.56974 does not exist on the remote host. Additionally the file name agent.56974 would indicate that agent forwarding would use a sshd process running with PID 56974 but there is not sshd running on that process. The other SSH environment variables also indicate that the ssh session was established using a client connection on port 50429 from my workstation. But running netstat | grep ssh shows

tcp        0    160 ansible:ssh             XXX.XXX.XXX.XXX:50432   ESTABLISHED

So it would appear that VSCode establishes a sshd connection to setup / start the server, but then closes that connection, leaving the environment that the server is running in inaccurate.

Would it be possible to use SSH multiplexing as a remedy to this problem?

@roblourens

This comment has been minimized.

Copy link
Member

commented May 3, 2019

Exactly correct @nathan-sain.

I would suspect that using ControlMaster would fix this but I see that the OP seems to have it set up.

@TomFrost

This comment has been minimized.

Copy link

commented May 3, 2019

@roblourens Thanks for the great explanation! I'd assumed a remote server was started with each new connection, but needing two connections to determine the forwarding port makes a ton of sense. The ugly side-effect is that two different machines making two different SSH connections to the same server instance can't each forward their own agent -- a bummer, since I do that with my office machine and a separate laptop. But given that solving that would require rearchitecture, accepting that only one agent can forward at one time seems sane.

With the above shortcoming accepted as a given, the solve for this could be as simple as writing SSH_AUTH_SOCK to a temp file that the server has an inotify watch on. But I suppose that's not as clean as an extension to the server process ;-)

@rcarmo

This comment has been minimized.

Copy link
Author

commented May 3, 2019

One iteration of my tmux scripts did exactly that (pull off and set SSH_AUTH_SOCK from a file, but inotify tends to get swamped in database servers and machines with lots of writes - and I often forgot to bother with sysctl -w a new max limit ;)

@roblourens

This comment has been minimized.

Copy link
Member

commented May 3, 2019

Using ControlMaster with the steps here does work for me:

https://code.visualstudio.com/docs/remote/troubleshooting#_enabling-alternate-ssh-authentication-methods

Meaning I can run ssh-add -l in a remote vscode terminal and see my local keys. First I had to kill the remote agent using the command "Kill VS Code Server on Host" to make sure we start a new one with the right environment. I think this would be my recommended workaround for now. I can add a note to the docs.

@rcarmo so you are using controlmaster for this connection? Can you check the value of SSH_AUTH_SOCK in a vscode terminal?

@rcarmo

This comment has been minimized.

Copy link
Author

commented May 3, 2019

Sure thing. Here's my Docker builder on my home LAN:

rcarmo@rogueone:~/Development/alpine-node$ set | grep SSH_
SSH_AUTH_SOCK=/tmp/ssh-sS32r23wcx/agent.31416
SSH_CLIENT='192.168.1.111 51626 22'
SSH_CONNECTION='192.168.1.111 51626 192.168.1.201 22'
rcarmo@rogueone:~/Development/alpine-node$ ls -al /tmp/ssh-*
ls: cannot access '/tmp/ssh-*': No such file or directory
rcarmo@rogueone:~/Development/alpine-node$ ssh-add -l
Error connecting to agent: No such file or directory

Either way, after killing the server using the "Kill VS Code Server on Host" I get pretty much identical results. But in my case every time I issue that command it seems to fail and I get a prompt to reload that VS Code window.

Oh, and the feature-request label is... suboptimal. General expectations are that this should work.

@rcarmo

This comment has been minimized.

Copy link
Author

commented May 3, 2019

A bit more experimentation yielded a positive result inside the VS Code terminal with:

    ControlPersist  600

Set in my Mac's ~/.ssh/config file. However, a 600 second timeout is frankly excessive, and leaving control sockets lying around tied to a daemon (even with nothing connected to it, and it being battle-tested, etc.) so I opted to set it to ControlPersist 5, which works.

I can now use ssh-add -l to commit manually via the terminal actually I can't but I did get the right key added to my agent - I was committing to the wrong git remote, which is my CI system and runs in the same box.

This is what I get in my terminal now:

rcarmo@rogueone:~/Development/alpine-python$ set | grep SSH_
SSH_AUTH_SOCK=/tmp/ssh-H2e7wJyIm8/agent.28181
SSH_CLIENT='192.168.1.104 57043 22'
SSH_CONNECTION='192.168.1.104 57043 192.168.1.201 22'
rcarmo@rogueone:~/Development/alpine-python$ ssh-add -l
2048 SHA256:2Fke2KiPZYa4JXdUXJsouBdex4wmcfHZ768Mfu2wLu4  (RSA)
rcarmo@rogueone:~/Development/alpine-python$ git push
bind: No such file or directory
unix_listener: cannot bind to path: /home/rcarmo/.ssh/sockets/git@github.com-22.WpGKA0tTMHVLwy5T
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Clicking on the "sync" button in VS Code fails with the following error:

Looking for git in: git
Using git 2.17.1 from git
> git rev-parse --show-toplevel
> git config --get commit.template
Open repository: /home/rcarmo/Development/alpine-python
> git status -z -u
> git symbolic-ref --short HEAD
> git rev-parse master
> git rev-parse --symbolic-full-name master@{u}
> git rev-list --left-right master...refs/remotes/origin/master
> git for-each-ref --format %(refname) %(objectname) --sort -committerdate
> git remote --verbose
> git check-ignore -z --stdin
> git pull --tags origin master
bind: No such file or directory
unix_listener: cannot bind to path: /home/rcarmo/.ssh/sockets/git@github.com-22.5YFf3eeSefYnNxZ3
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

That's the next step, I think, and will require the server component to invoke git with the right environment (so the terminal workaround, if it worked, would only be a partial solution).

incidentally, if anyone wants to test remote Node debugging on ARM32 containers, I'm doing builds for Raspberry Pis (armv6 and armv7) and can try adding ARM64 :)

@roblourens

This comment has been minimized.

Copy link
Member

commented May 3, 2019

However, a 600 second timeout is frankly excessive, and leaving control sockets lying around tied to a daemon (even with nothing connected to it, and it being battle-tested, etc.) so I opted to set it to ControlPersist 5, which works.

Oh, that makes sense. That will work but at some point the socket will time out and the next connection will create a new one, and the agent environment will be wrong after that point.

I don't quite understand where this is coming from: /home/rcarmo/.ssh/sockets/git@github.com-22.WpGKA0tTMHVLwy5T

Are you also using ControlMaster on your remote? Is that a path for your local machine for some reason?

@rcarmo

This comment has been minimized.

Copy link
Author

commented May 3, 2019

I'm not using ControlMaster on the remote - it has no SSH configuration whatsoever besides authorized_keys to allow me to log in.

But mind you, the socket is valid for as long as an existing connection is in place. Also, IIRC one of the reasons I didn't use ControlPersist is that it can get "stuck" due to a stalled process, and it broke my Ansible deployments (it was locked out of SSH until the timeout expired). This was a while back, can't remember the details.

As to the path, it is quite likely to be the default SSH socket path in Ubuntu, which means the environment isn't being propagated to terminal subprocesses.

@TomFrost

This comment has been minimized.

Copy link

commented May 3, 2019

Confirmed that I got this working with ControlMaster (Though Rob's link seems to be private to MS staff-- I believe he was pointing to https://code.visualstudio.com/docs/remote/troubleshooting#_enabling-alternate-ssh-authentication-methods).

It would be great to work on this a bit more though, as this solution wouldn't allow me to, say, close my laptop, drive home, and continue working without an exceptionally long timeout. Having the server able to update the SSH_AUTH_SOCK when it changes -- if adn only if it points to a nonexistent file -- would be ideal. That way multiple local machines connecting to the same remote remains usable as long as the first doesn't drop while the second is still connected. That's a solvable case in the future as well though.

@rcarmo

This comment has been minimized.

Copy link
Author

commented May 3, 2019

Confirmed that I got this working with ControlMaster (Though Rob's link seems to be private to MS staff-- I believe he was pointing to https://code.visualstudio.com/docs/remote/troubleshooting#_enabling-alternate-ssh-authentication-methods).

Well, I can read the original link, and saw nothing different from my config other than ControlPersist and the socket path - which shouldn't matter - but after a second read, it is the same path as the one that git complains about...

My takeaway right now is that the documented workaround in that link makes an assumption for ControlPath (that may not work out for everyone, even though it seems to match OpenSSH defaults - which I'm still researching)

@nathan-sain

This comment has been minimized.

Copy link

commented May 3, 2019

Unfortunately, Win32-OpenSSH does not support ControlMaster so for those of us suck running VSCode on Windows there does not seem to be a workaround at the moment.

@rcarmo

This comment has been minimized.

Copy link
Author

commented May 3, 2019

@nathan-sain Well, I use the WSL SSH client for everything, so maybe that is feasible (not near any Windows machines right now).

@rcarmo

This comment has been minimized.

Copy link
Author

commented May 3, 2019

Update: I created ~/.ssh/sockets on my Mac, set my ~/.ssh/config to the exact contents of the document (plus the Mac specific options for the keychain), and no, it doesn't work for me.

@jrobsonchase

This comment has been minimized.

Copy link

commented May 4, 2019

I've found that this works consistently for both ssh and gpg forwarding:

RemoteForward /run/user/2000/gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent
RemoteForward /run/user/2000/gnupg/S.gpg-agent.ssh /run/user/1000/gnupg/S.gpg-agent.ssh
SetEnv SSH_AUTH_SOCK=/run/user/2000/gnupg/S.gpg-agent.ssh

Since the socket is always in the same place, you don't have to worry about SSH_AUTH_SOCK only being valid for a single ssh session. It's not quite as "clean" as the ForwardAgent option, but it seems to get the job done.

@rcarmo

This comment has been minimized.

Copy link
Author

commented May 4, 2019

@jobsonchase those run paths are for Linux, right? I’m going to have a go at fiddling with SetEnv, but I have a feeling that fiddling with SSH configs risks breaking other tools (like Ansible, like I mentioned above).

@jrobsonchase

This comment has been minimized.

Copy link

commented May 4, 2019

@rcarmo Correct, specifically the flavors that have switched to using the XDG_RUNTIME_DIR rather than GNUPGHOME for the sockets. They're generally the ones that also start the agent with systemd --user, Arch in my case.

It's kind of unfortunate that there's no way (that I've found at least) to figure out the UID of the remote user to know where to put the sockets, or a way to specify the gpg-agent socket location like there is for the ssh-agent socket. You could put the S.gpg-agent.ssh (or whatever ssh agent socket you're actually using) in an easier-to-find location like /tmp/<username>/ssh-agent-socket, but I believe gpg removed the ability to specify a different socket path, so you're stuck putting it where it's expected to be.

On the topic of Ansible, I doubt this config would break anything - it's effectively the same as adding ForwardAgent yes to your config, just with a consistent socket path.

@singerb

This comment has been minimized.

Copy link

commented May 4, 2019

+1 for supporting agent forwarding; right now I commit on my remote sandbox machine with git CLI, since I have to run pre-commit hooks and it's easier to just use the environment there than to try and replicate it locally (getting a node_modules/ and .git/` folder synchronized with file sync tools is a nightmare). Being able to use the VS Code git UI to commit remotely and run the hooks in that environment would be great, but it relies on agent forwarding, naturally.

@benliddicott

This comment has been minimized.

Copy link

commented May 7, 2019

Workaround for Windows client, Linux server

I'm working around this as follows. @roblourens something similar may work for the product?

Connecting from code-insiders, I do not configure agent forwarding. Instead I have a single additional connection with agent forwarding (from a command prompt, but I suppose PuTTY would also work), and use a symbolic link to direct vscode to use that.

At the top of my .bashrc on the destination server:

if [ -z "$SSH_AUTH_SOCK" -o "$SSH_AUTH_SOCK" = "$HOME/.ssh/auth_sock" ] ; then
  # vscode connection
  # Direct vscode to use symlink, which we can set later.
  export SSH_AUTH_SOCK=$HOME/.ssh/auth_sock
else
  # agent forwarding connection. 
  # Direct agent symlink to the real port.
  ln -fs $SSH_AUTH_SOCK $HOME/.ssh/auth_sock
fi

It still has the shortcoming that it only allows a single agent forwarded to any destination.

Edit: I notice X11 forwarding also doesn't work, but using PuTTY and setting DISPLAY solves that.
Edit 2: Fixed case where it was occasionally overwritten

@ardikaveh

This comment has been minimized.

Copy link

commented May 10, 2019

ControlMaster method also doesn't work when connecting from multiple machines to the same remote server(work, home). I have to reboot the remote server for it to work again.

Why can't the terminal not automatically connect to the remote machine? If it behaved as it does without remote ssh there wouldn't be an issue.

@AgSync-Aaron

This comment has been minimized.

Copy link

commented May 12, 2019

I too am unfortunately running into this issue as well. I don't mean to pile on, I just don't have much to add to the conversation. If I can help test, or get you some other specific setup information I will.

When opening an ssh connection from a command prompt the agent forwarding works great. Just not through VS Code.


OpenSSH: OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
Version: 1.34.20-insider (user setup)
Commit: d6aa8eaf30c71df32d4543290675808440def630
Date: 2019-05-09T22:10:02.019Z
Electron: 3.1.8
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Windows_NT x64 10.0.17763

@Toxicable

This comment has been minimized.

Copy link

commented May 13, 2019

Also running into this issue I have this in my config file:

Host *
  ForwardAgent yes

Host LINUX_HOST
  User LINUX_USER
  HostName LINUX_HOST
  ForwardAgent yes
  LocalForward 127.0.0.1:4200 127.0.0.1:4200
  LocalForward 127.0.0.1:3000 127.0.0.1:3000

If I manually ssh ssh LINUX_HOST then I can use git on the other side - proving the agent is forwarded
If I remote with VSC then I am unable to use git

For now my work around is to use git in a seprate powershell window but it would be awesome be able to use git inside VS Code

@TomFrost

This comment has been minimized.

Copy link

commented May 14, 2019

@Toxicable and others who can't get this to work at all -- per the above, add these lines to your ssh config:

    ControlMaster auto
    ControlPath  /tmp/%r@%h-%p
    ControlPersist  600

Then in VSCode, Open the command palette (Cmd+Shift+P) and choose "Remote-SSH: Kill VS Code Server on Host".

A better solution (particularly one that supports multiple machines, like work + home) would be great, but for now this is an adequate workaround. It just requires that you re-kill the server if your connection drops for more than 10 minutes, or whatever your ControlPersist is set to.

From: https://code.visualstudio.com/docs/remote/troubleshooting#_enabling-alternate-ssh-authentication-methods

@rcarmo

This comment has been minimized.

Copy link
Author

commented May 14, 2019

I hesitate to recommend the ControlPersist workaround given its blocking behaviour. In fact, I disagree with it being a workaround unless it's set to something much shorter...

@benliddicott

This comment has been minimized.

Copy link

commented May 14, 2019

@TomFrost @Toxicable @AgSync-Aaron @rcarmo @ardikaveh @jrobsonchase
I have a workaround I am using daily, with no issues, see my comment above. I'm using it from Windows->Linux but there is no reason it shouldn't work for Linux->Linux or another combination.

@TomFrost

This comment has been minimized.

Copy link

commented May 24, 2019

Note that as of recently, the ControlMaster solution is no longer working for me. Perhaps a recent update changed that functionality?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.