Mosh 1.1.3 on OS X hangs on connect

[st3fan]

OS X 10.7.3. Brew. Installed with "brew install mobile-shell"

Connecting to an Ubuntu 10.04.4 LTS server results in a hang. Mosh prints one empty line after 'mosh foo.bar.com' and then nothing happens.

I see the following processes running:

 | | | \-+= 48073 stefan -zsh
 | | |   \-+= 73668 stefan perl /usr/local/bin/mosh foo.bar.com
 | | |     \-+- 73671 stefan ssh -S none -o ProxyCommand='/usr/local/bin/mosh' --fake-proxy -- %h %
 | | |       \-+- 73672 stefan perl /usr/local/bin/mosh --fake-proxy -- foo.bar.com 22
 | | |         \--- 73674 stefan perl /usr/local/bin/mosh --fake-proxy -- foo.bar.com 22

To make sure the server side is ok .. I can connect to this host from another Ubuntu machine.

[karlentwistle]

I am also experiencing this issue, however I can connect successfully to the host in another session.
OS X 10.7.3. Brew. Installed with "brew install mobile-shell" connecting to Ubuntu 10.04.4 LTS mosh 1.1.3

As st3fan said, mosh prints one empty line and hangs, it seems to be stuck on perl5.12

[adamgreig]

I killed a few of the (local) mosh-related processes on OS X and it suddenly started working and has been ever since. Not sure what exactly fixed it...

[simpx]

+1

[keithw]

Do you guys still have this problem if you change the first line of /usr/bin/mosh to read "#!/usr/bin/perl" ? That is, if you use the system Perl instead of whatever is first in your PATH?

[jackqu7]

I am having this problem. It's hanging on line 223 of /usr/bin/mosh which is waitpid $pid, 0;. The SSH process it's waiting for does not seem to terminate.

Commenting out this line does allow it the connection to succeed but obviously causes unterminating SSH processes to build up.

[masnick]

+1

I can confirm it still hangs with the /usr/bin/perl shebang.

[masnick]

It looks like running mosh user@server.com spawns three perl5.12 processes. If I kill two of them, mosh unhangs.

I still can't connect to my server (now mosh sticks on xx without contact forever). I'm not sure if this is a configuration issue or a result of killing those processes.

[rafaelperrone]

I'm having this "xx without contact" forever too and I didn't kill any process.

I'm trying to mosh from a Macbook Pro Lion (10.7.3) to a Linux Mint 12 desktop. Mint is behind a NAT listening to ssh on several ports (I can ssh to it using all these ports). I've opened UDP ports 60000-61000 too.

Installed mosh with the .pkg. It connects but then hangs with a blue first line saying:
mosh: Connecting... (xxx s without contact.) [To quit: Ctrl-^ .]

I don't know if this is the same problem as you are experiencing, but the error looks the same.

[st3fan]

@rafaelperrone It is not the same. You get the blue bar. I don't :-)

[rafaelperrone]

st3fan, you're right. Just tethered my phone connection and I could finally mosh to server. So it might be something related to my university's router blocking some ports. But which ports would it be? 60000-61000 UDP maybe?

NewAlexandria, maybe you have the same issue. Try to mosh using a different connection.

[st3fan]

Has anyone found a workaround for this issue? I tried the latest mosh version from github and it still has the same issue. Is there any more info that I can submit here to get more insight into what is causing this?

[st3fan]

Just a note that using the .pkg from mosh.mit.edu also gives me the same problem.

[keithw]

st3fan, it looks like the ssh command just isn't exiting for you when the mosh-server detaches. This is confusing (and I can't seem to reproduce it), but I appreciate your patience and let's try to work through it.

(1) What does "ssh -V" say?

(2) If you run "ssh -t SERVERNAME mosh-server", does that print the mosh-server startup message and then "[mosh-server detached, pid = 29095]" and then put you back at your local shell? (I.e. does SSH properly exit?)

(3) How about if you run this? "ssh -t SERVERNAME -o ProxyCommand="mosh --fake-proxy -- SERVERNAME 22" mosh-server" Does it put you back at your local shell?

[st3fan]

~ % ssh -V
OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011

~ % ssh -t foo.bar.com mosh-server

MOSH CONNECT 60001 AElSZE2La6CeDH13qkEonQ

mosh-server (mosh 1.1.3)
Copyright 2012 Keith Winstein <mosh-devel@mit.edu>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 581]
Connection to foo.bar.com closed.

Yup. The last command drops me back into the local shell.

[st3fan]

Ha! So I am using zsh. I switched from zsh to bash and now things work! I will try to find out if this is zsh related or profile related.

[st3fan]

Did a little more testing. When I start zsh with zsh -f to prevent it from loading my .zshrc it still has the same hanging issue. So it does not seem to be related to my specific profile.

[jm2]

Im having the same issue, but its not consistent, sometimes it'll happen and sometimes it'll succeed. I'm running bash.

[phik]

I appear to be having a similar problem -- but I believe it's actually a server issue in my case, not client.

I have two servers, both running Centos 5. Both are running identical mosh binaries.

When I run mosh-server via ssh to the working server, ssh exits immediately when mosh detaches.

When I do the same to the non-working server, ssh exits only after mosh-server times out and quits.

This happens regardless of which client I ssh from. It happens even when I run the ssh command on the non-working server itself:

$ ssh -t localhost mosh/bin/mosh-server

MOSH CONNECT 60001 wLjTiVHPj82mG9wis6NanQ

mosh-server (mosh 1.1.3a)
Copyright 2012 Keith Winstein mosh-devel@mit.edu
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 3725]
No connection within 60 seconds.

[mosh-server is exiting.]
Connection to localhost closed.

I use bash on all machines.

Commenting out the waitpid works around it, although at the risk of muddying the water of this bug, mosh really doesn't work very well when I do that. It drops keystrokes no matter how slowly I type, and when I exit mosh, my terminal is all screwed up (it no longer echoes anything until I do a "reset"). Neither of those symptoms occur on the originally-working server.

[phik]

(I should note that if I start the server and client manually, not only does it (obviously) duck the waitpid issue, but I don't have the dropped-character or terminal-screwup symptoms)

[st3fan]

Sorry to say but I don't think zsh is the problem. I tried to reproduce my results from last night and Icould not get it to work again. It would be nice if mosh had a --verbose mode. So that we can trace where it gets stuck.

[keithw]

phik and I worked to find a solution to his issue, which is fixed for him (on CentOS 5.0) by the above commit (0e9be1b), which needs to be applied to the mosh-server.

I'd be grateful if those of you with other configurations (e.g. Mac->Linux referenced by the original submitter) could also test this on the server side and let me know if it fixes the problem.

The basic diagnosis is that we were not detaching forcefully enough for some older versions of sshd.

[jm2]

The systems I was having the issue on were also running RHEL5 server-side, so I'll make sure to try the patch.

[st3fan]

@keithw That change does not fix this issue unfortunately. I removed the mosh package and installed the head version from this repository on both server and client. Same issues. I tried to connect 10 times. 2 out of 10 were succesful, 8 were hanging like I described in the first post of this issue.

[keithw]

Thanks for trying st3fan. Let's try to get to the bottom of this. Can you run:

ssh -t SERVERNAME 'perl -we "\$SIG{HUP} = q{IGNORE}; if ( fork() == 0 ) { close STDIN; close STDOUT; close STDERR; sleep 50 } else { exit }"'

Does that come back to your local shell immediately, or only after 50 seconds?

It seems to come back immediately for me (on Ubuntu 11.10) and phik (on CentOS 5).

[st3fan]

It returns immediately. I'm on Ubuntu 10.04.4 LTS.

[kmcallister]

It seems like #164 is an instance of this, where the client is on Cygwin.

[lhunath]

Especially annoying is that you can't ^C / ^\ / ^Z it, the only way out is to kill your terminal or dig through ps from another. The ctrl-^. also doesn't work. I don't think there should be any time during mosh's lifetime that it both ignores INT/QUIT/STOP and doesn't have alternative handlers installed.

[keithw]

st3fan, maybe we can arrange a time to be on the #mosh IRC channel together and nail this down, because it's hard to do without quick iteration and a reproducible test case. But please try the above commit (pushed to master), which tries to hew even closer to the Perl script that apparently works for you. Obviously I don't fully understand the conditions in which your SSH is not exiting.

[keithw]

lhunath, your point is well-taken. For what it's worth, while the mosh script is still running SSH, the SSH escape sequences (~. by default) should work. Obviously this is not the cleanest experience (how does the user know we are still somehow frozen without the SSH quitting?) but it does work.

[keithw]

Unfortunately commit cc8b1f2 seems to cause other problems, breaking the server completely, on OS X. Looking at this now.

[keithw]

All right, the above commit (on branch mac-wip) appears to paper over the problem for now. Please test if can; otherwise I will be back in a few hours and will debug more completely.

[simpx]

I compile and install it in Mac OS X 10.7.3, mosh to my Ubuntu 10.04 LTS server.It works!

[simpx]

oh, wait. I close the terminal, and mosh to server , It hangs on again :(

[keithw]

This fix from saurik (along with 18dc967, both now on master) seems to have fixed it for him. Would appreciate confirmation from others.

[st3fan]

Confirmation that master now works great for me. Ubuntu 10.04.4 LTS Server & OS X 10.7.3 Client.

[keithw]

Fantastic. Closing this one with relish.

[sartak]

This resolved the issue for me too. Great timing. :)