Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Mosh ignores .bashrc when starting remote server #465

Open
adamnovak opened this Issue Oct 15, 2013 · 13 comments

Comments

Projects
None yet
7 participants

I can start up mosh-server on the remote host with my approximation of what I think mosh is supposed to be doing:

[anovak@cruncher ~]$ ssh localhost mosh-server 'new' '-c' '8' '-s' '-l' 'LANG=en_US.UTF-8'
Server started without pseudo-terminal. Opening 80x24 terminal.

MOSH CONNECT 60001 U5/vyF3RQW7lpCOuUl4GNA

mosh-server (mosh 1.2.4a)
Copyright 2012 Keith Winstein <mosh-devel@mit.edu>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 31664]

I have mosh installed with a prefix, but my .bashrc (which runs for non-login shells like the above) adds it to my PATH.

However, mosh somehow is managing to invoke a remote bash that doesn't get the PATH defined in my .bashrc:

[anovak@cruncher ~]$ mosh --ssh="ssh -vv" localhost
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Executing proxy command: exec '/inside/home/anovak/.local/bin/mosh' --fake-proxy -- localhost 22
debug1: identity file /inside/home/anovak/.ssh/identity type -1
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file /inside/home/anovak/.ssh/id_rsa type 1
debug1: identity file /inside/home/anovak/.ssh/id_dsa type -1
debug1: loaded 3 keys
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.3
debug2: fd 5 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 131/256
debug2: bits set: 517/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'localhost' is known and matches the RSA host key.
debug1: Found key in /inside/home/anovak/.ssh/known_hosts:3
debug2: bits set: 517/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /inside/home/anovak/.ssh/identity ((nil))
debug2: key: /inside/home/anovak/.ssh/id_rsa (0x2b7885e50590)
debug2: key: /inside/home/anovak/.ssh/id_dsa ((nil))
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: An invalid name was supplied
Hostname cannot be canonicalized

debug1: An invalid name was supplied
Hostname cannot be canonicalized

debug1: An invalid name was supplied
Hostname cannot be canonicalized

debug2: we did not send a packet, disable method
debug1: Next authentication method: publickey
debug1: Trying private key: /inside/home/anovak/.ssh/identity
debug1: Offering public key: /inside/home/anovak/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug2: input_userauth_pk_ok: SHA1 fp 96:32:2f:c8:28:b2:c0:95:9a:bd:6b:17:b6:97:92:49:2a:4c:18:5b
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
debug2: fd 3 setting O_NONBLOCK
debug2: fd 6 setting O_NONBLOCK
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Entering interactive session.
debug2: callback start
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug1: Sending command: mosh-server 'new' '-c' '8' '-s' '-l' 'LANG=en_US.UTF-8'
debug2: channel 0: request exec confirm 0
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug2: channel 0: read<=0 rfd 3 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
bash: mosh-server: command not found
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 1 clearing O_NONBLOCK
Connection to localhost closed.
debug1: Transferred: stdin 0, stdout 0, stderr 33 bytes in 0.0 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 1869.9
debug1: Exit status 127
/inside/home/anovak/.local/bin/mosh: Did not find mosh server startup message.

The key line there is "bash: mosh-server: command not found".

Also, if I put an echo at the top of my .bashrc, it comes through when I run non-login ssh manually, but not when I try to connect with mosh.

I suspect that mosh is broken somehow and is telling the remote system to not start a proper non-login shell.

Member

andersk commented Oct 24, 2013

If mosh-server was not found, then mosh hasn’t had the chance to start any shells except the one started by ssh. The ssh command that mosh actually runs is

ssh -S none -o "ProxyCommand='/inside/home/anovak/.local/bin/mosh' --fake-proxy -- %h %p" -n -tt localhost -- "mosh-server 'new' '-c' '256' '-s' '-l' 'LANG=en_US.UTF-8'"

I’m assuming that command behaves differently than the one you ran. Maybe you can figure out why. I’m guessing that the -n and/or -tt interacts with your personal dotfiles somehow—it’s not uncommon for dotfiles to include conditional tests such as [ -t 1 ].

lzap commented Nov 1, 2013

I cannot confirm that. Works for me in Fedora.

Where is the documentation for the -n and -tt options being used here? I
cannot find them
in the bash man page, and the ssh man page only lists -n, which means it
should use /de/vnull as stdin.

How would this interact with dotfiles. My dotfiles do indeed include some
conditional tests, but I don't see anything checking [ -t 1](i.e. if stdin
is a terminal).

Could someone post working dotfiles?

On Fri, Nov 1, 2013 at 9:20 AM, Lukáš Zapletal notifications@github.comwrote:

I cannot confirm that. Works for me in Fedora.


Reply to this email directly or view it on GitHubhttps://github.com/keithw/mosh/issues/465#issuecomment-27577934
.

Contributor

glance- commented Nov 1, 2013

The -n and -tt options ain't options to bash, its options to ssh.

Its up to your server to handle your .dotfiles, and mosh doens't care if your login shell is bash, tcsh or something else.

You probably have some weird if-case checking something in your .bashrc or your global bash startup.

I've pared my dotfiles down to the following:

-bash-3.2$ cat .bashrc
echo ".bashrc running"
export PATH=$HOME/.local/bin:$PATH
export LD_LIBRARY_PATH=~/.local/lib:$LD_LIBRARY_PATH

-bash-3.2$ cat .bash_profile
echo ".bash_profile running"
export PATH=$HOME/.local/bin:$PATH
export LD_LIBRARY_PATH=~/.local/lib:$LD_LIBRARY_PATH

I believe the problem is the -tt option to ssh:

-bash-3.2$ ssh localhost "echo \$PATH"
.bashrc running
/inside/home/anovak/.local/bin:/usr/local/bin:/bin:/usr/bin
-bash-3.2$ ssh -tt localhost "echo \$PATH"
/usr/local/bin:/bin:/usr/bin
Connection to localhost closed.

When ssh is asked to allocate a tty, neither of the two dotfiles defining the $PATH to include mosh-server is run.

I am now looking into whether this is expected behavior for the ssh -tt option, and how it might be worked around.

Does ssh -tt execute dotfiles on your system?

Normally ssh -t means "allocate a pty on the server side", which usually
means "run a login shell". ssh -tt means only "and allocate a pty even if
the client itself has no local controlling pty".

If you run a command then the server might not run a login shell. What you
have to do here is look at your server's documentation and configuration.
This is not a client-side issue.

Member

andersk commented Nov 5, 2013

Yes, on my system (Ubuntu), and I expect most others,

  • ssh -tt with no other arguments runs the first of ~/.bash_profile, ~/.bash_login, or ~/.profile that’s readable (and the default ~/.profile runs ~/.bashrc if $BASH_VERSION is nonempty), and
  • ssh -tt COMMAND runs ~/.bashrc.

What is your server running? Is it possible that your default shell is not actually /bin/bash (try getent passwd anovak)? Do you have a ~/.ssh/rc or /etc/ssh/sshrc file?

The server I am connecting to runs RHEL 5. My default shell is indeed bash. ssh -tt COMMAND does not appear to run ~/.bashrc on this server. On Ubuntu, ssh -tt COMMAND does run ~/.bashrc. Apparently, not running ~/.bashrc is really the standard behavior; for a non-interactive, non-login shell, only the file pointed to by the $BASH_ENV variable is supposed to be sourced. See for example http://shreevatsa.wordpress.com/2008/03/30/zshbash-startup-files-loading-order-bashrc-zshrc-etc/ and https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html. Many distributions decide that this standard behavior is quite silly, and turn on the SSH_SOURCE_BASHRC compile-time option for their bash build, to force sourcing of .bashrc on all shells run by ssh. (See http://lists.gnu.org/archive/html/bug-bash/2008-10/msg00052.html and what may or may not be Debian's patch that accomplishes this at http://stuff.mit.edu/afs/sipb/project/linerva/no-backup/root/bash-2.05b-2/debian/patches/deb-bash-config.dpatch).

Unfortunately, there is by default no way (as far as I can tell) for a normal user to set $BASH_ENV for a non-interactive non-login shell invoked over SSH; environment variables defined in ~/.ssh/rc don't seem to get exported to the shell, and ~/.ssh/environment is not used under the default OpenSSH settings.

Is there a way to instruct mosh not to pass the -tt option to ssh? If it didn't, would that break mosh-server?

Member

andersk commented Nov 6, 2013

Aha, that’s what I’d forgotten about. I think you now have a correct diagnosis here.

Note that the shell that would later spawned by mosh-server is a login shell that reads all the standard dotfiles, so if you just pass mosh --server=/inside/home/anovak/.local/bin/mosh-server localhost, things should work.

The -tt was added in #378. It’s possible we may want to replace that with some other strategy, due to this and perhaps #464.

Contributor

glance- commented Nov 6, 2013

When/if we can assume that the mosh-server have 45bba44, we can drop the -tt part. If you try to run without -tt against a mosh-server from before that, it will die on a assertion.

Member

andersk commented Nov 6, 2013

Okay. 45bba44 is not part of any released Mosh version yet, so that will be a while…

Same here (also a RHEL5) server). I cannot start the server with --server because I need to set other paths and librariers to make mosh-server work.

Manually patching /usr/bin/mosh (placing a copy in ~/bin) to drop the "-tt" makes it work.

clacke commented Apr 8, 2016

I cannot start the server with --server because I need to set other paths and librariers to make mosh-server work.

In case someone reads this later on, one solution here is to use --server to point to a wrapper that sets up those paths and libraries, then runs mosh-server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment