mosh script interacts badly with ControlMaster

[ebroder]

I'm not sure exactly what's going on, but I use ControlMaster to avoid establishing multiple connections to a server. If I have an open ssh connection to a server (and therefore an active ControlMaster socket), starting mosh fails:

evan@warwick:~$ ssh -N broder@spock.tribbl.es

evan@warwick:~$ ls .ssh
authorized_keys                    id_dsa      known_hosts
authorized_keys2                   id_dsa.pub  known_hosts.old
config                             id_rsa
control-broder@spock.tribbl.es:22  id_rsa.pub
evan@warwick:~$ mosh broder@spock.tribbl.es
/usr/bin/mosh: Did not find mosh server startup message.

The relevant portion of .ssh/config is:

Host *
    ControlPath ~/.ssh/control-%r@%h:%p
    ControlMaster auto
    ControlPersist 1

[keithw]

Thanks, nice catch. The problem seems to be that ControlMaster overrides ProxyCommand, which we use (since ab29746) to detect the remote IP address so that mosh-client can connect to the same host we're SSHing to.

One fix is that we just give "-S none" to ssh, which will disable ControlMaster on the Mosh startup sessions while leaving it in place for any other sessions. Is this going to be annoying (i.e. are you depending on ControlMaster to not have to retype your password, or just for efficiency)?

Possible fix 2 is that we fall back in this case to connecting to the server's idea of its IP address (instead of the client's idea of its IP address that we get from ProxyCommand). This means that you will be able to set up a Mosh connection over ControlMaster, except when the server is behind a NAT (which works currently).

Thoughts?

[ebroder]

I'm only using ControlMaster to short-circuit the connection process, so disabling it for mosh connections would be fine for my use case.

[keithw]

Ok, I will just add -S none to the mosh startup script then.

[lilydjwg]

This fix annoys me a lot, because establishing a new ssh connection is slow for me (luckily I do not need retype password for the most time). I usually keep a ssh connection to transfer files. If it were quick, I might just use ssh after all :-)

I'd like to have an option to disable this behavior.

[ScoreUnder]

This fix annoys me a lot

You're not the only one. I have a workaround though:

#!/bin/sh
die() { printf %s\\n "$*" >&2; exit 1; }

# Start mosh on the server, retrieve its IP from ipinfo.io
# Feel free to replace with a better method of finding the IP
info=$(ssh "$@" <<\END_SSH
    . /etc/locale.conf >&2;
    export LANG
    env -u SHLVL mosh-server || exit
    echo '=cut='
    curl -sS ipinfo.io/ip
END_SSH
) || die 'SSH reported failure, giving up.'

# Ensure these variables are not already filled (in the environment)
# Because we check their contents later
mosh_port=
MOSH_KEY=

while read -r mosh connect port key; do
    # Don't allow curl output to bleed into mosh output parsing
    [ "$mosh" = '=cut=' ] && break

    if [ "$mosh $connect" = "MOSH CONNECT" ]; then
        mosh_port=$port
        MOSH_KEY=$key
        break
    fi
done <<EOF
$info
EOF

if [ "$mosh $connect" != "MOSH CONNECT" ] || [ -z "$mosh_port" ] || [ -z "$MOSH_KEY" ]; then
    die 'Did not find mosh server banner'
fi

lf='
'
# IP should be the very last line, so delete everything up-to-and-including the final LF
ip=${info##*"$lf"}

# Only really need to check for "-" which could be used to pass switches to mosh-client
# Anything else mosh-client will reject as a bad IP
[ "${ip#-}" = "$ip" ] || die 'Malformed IP from ipinfo.io'
# If ipinfo didn't report anything, the last line will be our '=cut='
[ "$ip" = '=cut=' ] && die 'No IP from ipinfo.io'

export MOSH_KEY
exec mosh-client "$ip" "$mosh_port"

Judge for yourself how much of a hack that is.
I've tried my best to keep it secure in the sense that a MITM between the server and ipinfo.io shouldn't cause any security issues, but of course they can still cause a denial of service because the script might no longer work.

If your server is not behind a NAT, you can replace the curl line entirely with something like printf %s\\n "$SSH_CONNECTION" | awk '{print $3}', similar to how #716 proposes to solve the problem of when ProxyCommand is unavailable.

Depending on how you feel about SSH environment variables you may want to add SSH_CLIENT and SSH_CONNECTION to the list of variables to unset in the env command near the top of the script.

[lilydjwg]

@ScoreUnder Thanks! 👍 Your script works well and doesn't need a JSON parser actually. Just curl ipinfo.io/ip and you'll get it :-)

[ScoreUnder]

Thanks for the catch, I've now updated the post above to reflect that. It's always nice to be able to remove unnecessary dependencies 😄

[jettero]

This was closed in 2012?

I just found that -S none in the mosh client script. sigh There's got to be a better way to do that.

[cgull]

Actually, now that we have --experimental-remote-ip=local and --experimental-remote-ip=remote, we could remove -S none for those options.

(Compromises, compromises. There's no perfect solution to finding the remote IP.)

[cgull]

Oops, I commented before looking at the code-- which already does exactly that. Try those options.

[jettero]

Oh, interesting. In my mosh-client "bin" there's a hardcoded -S none ... you're saying if I update there's commandline args that disable it I think. Checking.

Oh, apparently it's only "hardcoded" if $use_remote_ip is "proxy". Apparently I can use --experimental-remote-ip local, remote, or proxy. Proxy being -S -oProxyCommand=blah ... Interesting. Oh, lol, you even said that a comment before last.

[lilydjwg]

Sadly --experimental-remote-ip doesn't work for me in some cases. With remote, it gets the intranet IP; with local, it doesn't pick up my settings in ssh config.

[jettero]

What about some of these environment variables?

ssh to outside IP of a host, then issue set | grep ^SSH_:

SSH_AUTH_SOCK=/tmp/ssh-xxxxxx/agent.2471
SSH_CLIENT='75.x.x.x 41434 22'
SSH_CONNECTION='75.x.x.x 41434 10.x.x.x 22'
SSH_TTY=/dev/pts/10

ssh to inside IP (through VPN) of the same host set | grep ^SSH_:

SSH_AUTH_SOCK=/tmp/ssh-xxxxx/agent.28706
SSH_CLIENT='10.x.x.x 45992 22'
SSH_CONNECTION='10.x.x.x 45992 10.x.x.x 22'
SSH_TTY=/dev/pts/10

[lilydjwg]

@jettero In my case the server is behind a NAT with port-forwarding settings. The server itself doesn't know its outside IP. If --experimental-remote-ip=local could pick up my ssh_config settings it would work well for my case.

I know this option resolves the hostname itself and passes IP to ssh in case the hostname resolves to a different IP. My hostname won't do that however, and I think everyone should ensure the hostname they're sshing into doesn't change back and forth frequently.

[jettero]

Yeah, mine seems to know. If I ssh to it on the outside IP, even through the nat, the env vars show the pub IP and the inside IP both. See the first block. Oh, I suppose that's false actually... hrm.

[ssb22]

I have an account on an SSH server run by an institution which stores users' home directories on a Windows filestore and sets their Linux SSH server to mount them via CIFS. Unfortunately this CIFS arrangement is such that the SSH server requires a user's password before it can access that user's home directory, therefore .ssh/authorized_keys is not an option and a password must be typed. So I set a ControlMaster to avoid having to re-type my password into every single ssh and scp command. The -S none fix means I do have to retype my password when running mosh. But thankfully, adding --experimental-remote-ip=remote to my mosh alias works for me because the server knows its outside IP.