New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't use a ProxyCommand-based SSH connection #285

Open
inducer opened this Issue Jun 6, 2012 · 39 comments

Comments

Projects
None yet
@inducer

inducer commented Jun 6, 2012

The NAT/firewall-busting problems of issue #48 notwithstanding, mosh also has issues with the "ProxyCommand" style of connecting to machines behind NATs. For specificity, here's an example from my .ssh/config:

Host cims
  HostName linax1.cims.nyu.edu
Host mauler
  User kloeckner
  ProxyCommand ssh cims nc mauler 22

This is convenient, because it lets me type ssh mauler, and ssh will connect to linax1.cims.nyu.edu, and the 'outer' ssh run will then run over the tunnel thus set up. I understand this is hard for mosh to imitate, given UDP connections and what not, so let's assume that there's a hole poked into the firewall at port 60000 UDP on linax1.cims.nyu.edu that lets me get to mauler.

The main problem then is that mosh tries to resolve mauler locally to find the host to connect to, which of course doesn't succeed. I'd argue that it could potentially be smarter about finding the public IP of the target host--when it starts, it is executing codes on both ends of the connection after all. Failing that, I'd like to hhave an option to specify which host is actually meant.

@keithw

This comment has been minimized.

Member

keithw commented Jun 11, 2012

You're right that you can't use a ProxyCommand, but the reason is even simpler. Mosh doesn't try to resolve your hostnames -- it lets SSH do that. You can have a host alias in your SSH configuration and that will work fine. However, mosh itself uses the ProxyCommand feature (because it needs to know the IP address it's connecting to, in case there are multiple A records).

If you can find a way to start up the mosh-server on the remote end, and you can give the mosh-client an IP address to connect to, that's really all you need.

@jonesmz

This comment has been minimized.

jonesmz commented Jul 31, 2012

My use case is far more complex than inducer's, but if mosh supported ProxyCommand, it would make mosh a drop in solution.

Here's a snippet of my ssh config

host home
ProxyCommand connect -H proxy:3128 %h 443

host home-desktop
ProxyCommand ssh -q home nc %h %p -w 30 2> /dev/null

As you can see, I'm using ssh over a https proxy to form a tunnel to my home gateway, and then using that tunnel to connect to my home desktop machine.

The https proxy server that we have at my day job is unbelievably unreliable, disconnecting several times an hour, but it's the only way to punch out of the building.

I use autossh to keep a psuedo-persistent connection open, but still have to deal with my existing session being interrupted.

If mosh could be set up to pass its data through the ssh session used to start it (or a new one created with the same parameters), then mosh would be a viable option for me when at work.

Thanks

@nicolai86

This comment has been minimized.

nicolai86 commented Aug 24, 2012

+1 on this, I'd greatly appreciate ProxyCommand support.

My setup is simpler tho, support for something like this:

 Host foo

 Host bar
   ProxyCommand ssh foo nc %h %p

would be enough for my needs - and make mosh a drop in solution as well.

@jpawlowski

This comment has been minimized.

jpawlowski commented Jan 6, 2013

+1

@tribut

This comment has been minimized.

tribut commented Apr 1, 2013

If you need ProxyCommand to connect a host but it can be reached from the internet via UDP, you could use a script like this. It starts mosh-server on the remote machine (using SSH and the ProxyCommand from .ssh/config), parses the output and then connects using mosh-client to the machine directly.

@grooverdan

This comment has been minimized.

grooverdan commented Apr 26, 2013

I guess you could use mosh on the hop if that the only place that has access to farside.

host hop
host farside
  ProxyCommand mosh -- hop  -W farsideaddr:22

(-W is sometimes better option than netcat.)

Instead of using ProxyCommand to determine the address mosh could use the LD_PRELOAD environment variable, overwrite/proxy the connect system call method (to the SSH server) and use that resolution result to determine/resume the specific connection of hop.

@andersk

This comment has been minimized.

Member

andersk commented Apr 26, 2013

@grooverdan: mosh cannot be used as a ProxyCommand. That wouldn’t make any sense. The job of a ProxyCommand is passing on the raw, encrypted SSH stream; mosh doesn’t work that way.

You can, however, run mosh hop ssh farside, ssh hop mosh farside, or even mosh hop mosh farside, to use mosh for one or both legs of the connection. This obviously requires that you trust hop with the credentials you use to access farside.

@Daviey

This comment has been minimized.

Daviey commented Jun 23, 2013

I make quite heavy use of bastions, where the destination (farside) is not guaranteed to have mosh installed.

$ mosh hop ssh farside , seems to provide the benefits of mosh - but in a generic environment. I wondered if anyone has integrated this into their ssh config, to make the process simpler.

@porterjamesj

This comment has been minimized.

porterjamesj commented Sep 23, 2013

+1 for support of something like this, if its possible. I ssh into a bunch of VMs (farside) unaccessible from the Internet via a login node (hop) which is, but my situation is the opposite of @Daviey's. I have root on the VMs and so I can install mosh there, but I can't install it on the hop. Is there any way to get the benefits of mosh in this situation?

@felipebalbi

This comment has been minimized.

felipebalbi commented Mar 20, 2014

+1 for this, just figured out I can't use mosh from work to my server due to mosh ignoring ProxyCommand from my .ssh/config.

@akalin

This comment has been minimized.

akalin commented Jul 12, 2015

I came up with a potential solution that avoids ProxyCommand, but has its own downsides. There is a LocalCommand directive that causes ssh to execute a command on the localhost when the connection is established, with the same substitution rules (e.g. %h => remote hostname) as ProxyCommand. Of course, any existing LocalCommand directive will be clobbered, but I think that one is less likely to be used.

The downside is that LocalCommand requires the PermitLocalCommand=true option to be set, too, and that one enables local command execution via the !command escape sequence in ssh(1). So this has security implications -- can an exploit on the remote host use mosh-server to run arbitrary commands on the connecting host? I did a quick search, but couldn't find any history on the LocalCommand and PermitLocalCommand options, but it smells like something that was originally permitted by default, but locked down to avoid "surprise" shell execution vectors.

Anyway, my changes to master are master...akalin:avoid-proxy-command , and my changes to the stable version (which I tested and works swimmingly) are mosh-stable...akalin:avoid-proxy-command-stable .

@akalin

This comment has been minimized.

akalin commented Jul 12, 2015

Oh, and I guess another upside to LocalCommand is that it avoids the need for the "fake proxy" used in ProxyCommand -- one fewer process and one fewer source of buffering to worry about!

@akalin

This comment has been minimized.

akalin commented Jul 12, 2015

Another possible solution that avoids both LocalCommand and ProxyCommand, but is more brittle -- run ssh with -v, and parse the verbose output to look for the line:

debug1: Connecting to x.x.x.x [x.x.x.x] port 22.

@andersk

This comment has been minimized.

Member

andersk commented Jul 13, 2015

@akalin: Your LocalCommand proposal will cause the hostname to be resolved to an IP address separately by ssh and by mosh. They could get different results, either in the case of a DNS round-robin setup, or if there are IPv6 addresses involved.

Your -v proposal doesn’t solve the problem: if a user passes their own ProxyCommand, then ssh does not print such a debug message.

@cgull

This comment has been minimized.

Member

cgull commented Jul 13, 2015

For some users (such as @akalin) this is a worthwhile tradeoff, obviously. We might want to support both methods.

Once OpenSSH has their library API available, using that and then examining the socket it creates may be a cleaner solution to all of this. That partly depends on how elegant that API turns out to be and how much code is needed to make it all go. There are various ssh libraries already, of course, but they vary in support for ssh_config and its options, and in quality of implementation.

@akalin

This comment has been minimized.

akalin commented Jul 13, 2015

@andersk ah, you're right on both counts. But yeah, as @cgull said, maybe it should be an option, until a cleaner solution can be found.

I have a vague idea of getting the pid of the ssh process and using lsof/netstat to find the ip address, but that adds extra dependencies and is more complicated.

@cgull

This comment has been minimized.

Member

cgull commented Jul 13, 2015

Another thought: if you resolve a hostname and you get a single A or AAAA record with a long TTL, and no intermediate CNAME, you could heuristically decide that LocalCommand is safe, and you might be right most of the time.

@andersk

This comment has been minimized.

Member

andersk commented Jul 13, 2015

We rely on ssh to map the provided hostname to an actual hostname (which may be changed by ssh_config), so we don’t get to make that choice until after we have already decided to use ProxyCommand or LocalCommand.

@ekacnet

This comment has been minimized.

ekacnet commented Jan 4, 2016

I was looking at this issue because I have the same needs.
Indeed it's ssh who make the translation from the name specified on the command line to the real hostname/ip but still there is a risk that the fake proxy will resolve the name to a different one than the one that was resolved by ssh.

@andersk

This comment has been minimized.

Member

andersk commented Jan 4, 2016

@ekacnet, no, there’s no such risk: when using a ProxyCommand, ssh only performs the alias translations specified by ssh_config and does not do any DNS resolution itself.

@ekacnet

This comment has been minimized.

ekacnet commented Jan 4, 2016

@andersk I know that ssh_config is not doing the DNS resolution but ssh is doing a DNS resolution from myhost.mydomain.com to IP w.x.y.z.
And the proxy command is also doing a resolution.

The thing I missed is that when using proxycommand, the ssh command is reading everything from the stdout of the proxycommand.

@andersk

This comment has been minimized.

Member

andersk commented Jan 4, 2016

@ekacnet I understand your claim, and I’m telling you that it’s incorrect. ssh does not do a DNS resolution from myhost.mydomain.com to w.x.y.z when using a ProxyCommand.

@studgeek

This comment has been minimized.

studgeek commented Apr 12, 2016

In our corp environment we use a ssh ProxyCommand handler to handle certain security handshake aspects. Would be great if I could just drop that into mosh also.

@cgull

This comment has been minimized.

Member

cgull commented Apr 12, 2016

Please try Mosh master if you can; I've recently added an experimental change that will help some cases (though if firewalls or NAT are involved, probably not).

@fluidmindorg

This comment has been minimized.

fluidmindorg commented Apr 20, 2016

Thanks @cgull . Can you tell us what the experimental change is, and how it works?

@cgull

This comment has been minimized.

Member

cgull commented Apr 21, 2016

The change is to send a snippet of shell commands to the server that echo the SSH_CONNECTION variable to the client, and adding some code in the Mosh perl script to interpret that info. However, I've decided that doing that automatically/always is a bit risky and difficult to support, and I'm working on moving it to an option, along with some other new features. Coming soon to a Github repo near you.

@packeteer

This comment has been minimized.

packeteer commented Apr 27, 2016

I'm using Corkscrew to CNTLM to the web. It's messed up, but that's the nature of corporate workpsaces

@tysonclugg

This comment has been minimized.

tysonclugg commented May 5, 2016

How about providing some new command line behaviour:

  1. --ip=ADDR: IP address (v4 or v6) supplied, don't use ProxyCommand.
  2. --host=FOO: Resolve FOO to determine remote IP, don't use ProxyCommand.
  3. --resolve: Resolve host to determine remote IP, don't use ProxyCommand.
  4. No arguments: Use current ProxyCommand based implementation.

All resolving should be done via getaddrinfo syscall, which honours /etc/hosts. I'd like if --resolve eventually becomes the default, perhaps a --no-resolve option might be introduced that preserves the current behaviour.

@cgull

This comment has been minimized.

Member

cgull commented May 9, 2016

I just added an --experimental-remote-ip option to Mosh master, which adds two new ways for Mosh to handle address resolution without using ProxyCommand:

  • Obtain the address from the server
  • Resolve the hostname to an address before passing it to ssh or mosh

They'll each handle some use cases, but all three methods have limitations, and I'm sure they won't help some people. Additionally both of these new methods make it more possible to use non-OpenSSH clients (dbclient works but complains about options it doesn't support). Please try this and let us know how it works or doesn't for you.

@tysonclugg

This comment has been minimized.

tysonclugg commented May 24, 2016

Adding options to interrogate the remote host to determine it's IP is good, but in reality what you are trying to achieve is NAT traversal. Have a read of https://en.wikipedia.org/wiki/NAT_traversal and consider what can be done using ICE/STUN/etc to establish a connection. Mosh should not be inventing new NAT traversal techniques.

@cgull

This comment has been minimized.

Member

cgull commented May 24, 2016

Actually, no. We're quite well aware of the issues with server-side NAT traversal, and these changes aren't intended to help with that. They're intended to help with the difficulty of obtaining the remote address reliably from the ssh connection, so that ssh and Mosh traffic go to the same server.

I'm working on a moshd to help with issues with port usage on servers, and this shows some promise of being extensible to help with server-side NAT problems.

Personally, I'm not a big believer in NAT; I believe VPNs or IPv6 are generally better solutions. But there's certainly many potential use cases for Mosh that don't involve either.

@jonesmz

This comment has been minimized.

jonesmz commented May 25, 2016

If you run a full ICE RFC 5245 session, or even better ICE-Trickle https://tools.ietf.org/html/draft-ietf-ice-trickle-01, you'd be able to reliably enumerate all local and remote ip addresses, as well as respond to changes in them.

@andersk

This comment has been minimized.

Member

andersk commented May 25, 2016

Server-side NAT traversal is tracked as #48, not here.

@craftyguy

This comment has been minimized.

craftyguy commented Sep 17, 2016

This bug pretty much makes Mosh useless for me and my corporate colleagues who are trying to connect out from behind a proxy.

@andersk

This comment has been minimized.

Member

andersk commented Sep 17, 2016

@craftyguy Do you have an unusual setup that requires proxied SSH connections but allows unproxied UDP connections? If so, you’ve found the right issue and the new --experimental-remote-ip=local or --experimental-remote-ip=remote options in Mosh 1.2.6 may work for you.

If not, then this issue is not what makes Mosh useless for you. This is not a catch-all issue for connecting from behind arbitrary proxy servers. Mosh requires UDP, and there’s no getting around that. (There’s some discussion about TCP support over at #13, but it’s unlikely to be implemented any time soon.)

@kamushadenes

This comment has been minimized.

kamushadenes commented Oct 12, 2018

I use assh, which adds ProxyCommand to every connection, and I got it fully working with the following steps:

  • Create /usr/local/bin/mosh_fallback
#!/usr/bin/env bash
mosh --experimental-remote-ip=remote "$@"

status=$?

if [ $status -eq 5 ] || [ $status -eq 127 ] || [ $status -eq 10 ]; then
        ssh "$@"
fi
  • Just alias ssh with alias ssh "/usr/local/bin/mosh_fallback"

After that, it worked flawlessly on all my use cases, automatically falling back to ssh when mosh isn't installed on the remote.

@sequencer

This comment has been minimized.

sequencer commented Oct 19, 2018

I use assh, which adds ProxyCommand to every connection, and I got it fully working with the following steps:

* Create /usr/local/bin/mosh_fallback
#!/usr/bin/env bash
mosh --experimental-remote-ip=remote "$@"

status=$?

if [ $status -eq 5 ] || [ $status -eq 127 ] || [ $status -eq 10 ]; then
        ssh "$@"
fi
* Just alias ssh with `alias ssh "/usr/local/bin/mosh_fallback"`

After that, it worked flawlessly on all my use cases, automatically falling back to ssh when mosh isn't installed on the remote.

I can use mosh trying to connect my server, however the it says

/usr/bin/mosh: Using remote IP address 172.25.23.30 from $SSH_CONNECTION for hostname quilava-proxy

mosh did not make a successful connection to 172.25.23.30:60001.
Please verify that UDP port 60001 is not firewalled and can reach the server.

(By default, mosh uses a UDP port between 60000 and 61000. The -p option
selects a specific UDP port number.)
[mosh is exiting.]

172.25.23.30 is my internal IP address,
here is my ssh_config

Host some-server-behind-proxy
   Hostname 172.25.23.30
   ProxyCommand nc -x localhost:1080 %h %p

will it be a problem with my nc command?

@smaslennikov

This comment has been minimized.

smaslennikov commented Nov 2, 2018

I'm getting similar issues when I set RemoteCommand in .ssh/config. Is this the same issue, or should it be tracked elsewhere?

@andersk

This comment has been minimized.

Member

andersk commented Nov 2, 2018

@smaslennikov It’s similar but even more fundamental. Mosh uses the remote command to launch the remote mosh-server. (It uses the command argument instead of the RemoteCommand option, but SSH doesn’t allow you to use both, so SSH errors “Cannot execute command-line and remote command.”)

If you want to specify the command to launch inside the Mosh session, simply provide it after -- on the Mosh command line: mosh HOSTNAME -- screen -dr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment