Reconnection and garbage collection

[keithw]

Allow reconnection to existing sessions that were somehow severed. Give command to clean up disconnected sessions.

[pimlott]

I just started using mosh and this is my top priority wish. screen and dtach are not as solid as mosh and disrupt the great user experience that mosh offers.

I suggest simply allowing the user to specify a session name on the command line ("mosh --session-name dev1 ..."). When a session name is given, mosh-server looks for and reconnects to an existing session before starting a new one. Of course there are a lot more details, but you can figure them out. :-)

[pimlott]

I thought this would be as simple as a wrapper that would run a new mosh-client if the appropriate server were already running. But this does not work. If you run mosh-server and mosh-client by hand, then kill -9 mosh-client and run it again, it cannot connect. So I guess this has to be fixed at the protocol level. :-(

I have a prototype wrapper script ready for when this is fixed. By the way, one thing you might want to change is that mosh-server prints its port and key or stdout, but its PID on stderr. Since the wrapper script needs both, it has to do extra work to read both outputs. It would be simpler to print all normal messages to stdout.

[kmcallister]

As @joneskoo pointed out in #284, we could at least add something to kill any mosh-server currently listening on the same port. This would help firewalled users who open only one port for Mosh.

[phmarek]

Any progress on the "re-connect to existing server" front?

[kmcallister]

There are a few possible designs for reconnecting:

• Log in with SSH again, and communicate with the old mosh-server (over a UNIX domain socket or something) to set up a new cryptographic session
• Have mosh-client save its cryptographic state (session key and nonce) to disk, so that a new client can resume the session.

Either design adds a significant amount of complexity and security attack surface. In particular, using the same nonce and key more than once is fatal to security. So in the second design, we would need to be very careful to never use an even slightly stale state file — I consider this basically impossible to get right. @pimlott: you should be careful too, if you go back down the route of launching a new mosh-client with the same key.

A major design goal of Mosh is to keep the security story as simple as possible. I don't think reattaching is a compelling enough feature to go against this principle, when we have acceptable workarounds like mosh host -- screen -dR. (I say this as someone who uses this workaround daily.)

However, I do think there are some simpler features we could implement to make these workarounds nicer. See #324 for an example.

[phmarek]

What's the attack surface for the first item? If I can get to the machine, I can just start GDB and get any cryptographic state out of the running process. So the mosh-server might accept a short plain-text exchange on its UDP socket (sent from localhost) to re-initialize it's crypto data (or on a separate unix-socket - but the requirement "same machine" could be solved by using a mmap()ed file for communicating the new state, and using UDP only to start the re-initialization, eg. by telling the mmap() filename).

In fact, using GDB (and exported variables for the counter and the crypto-state) might be a way to make it work from the outside without any more work within mosh ;)

[kmcallister]

@phmarek: It's mainly that we need a place to put those sockets on the filesystem, such that only the user can access them, and this needs to be correct for all UNIXes (which don't agree on interpreting permissions bits on socket files) and also correct for weird filesystems like AFS which implement their own access control model. I'm not saying this is impossible — programs like screen have to do something similar. But it's just another thing that we could screw up.

(Actually, I think screen uses its setgid powers to create sockets or pipes in an area not normally accessible to users. setuid/setgid is of course another huge can of worms we don't want to open.)

On the other hand, accepting an unauthenticated message with a user-specified (i.e. attacker-controlled) filename and then opening and writing to that file seems like a really bad idea...

Right now, Mosh doesn't do anything with the filesystem (except print /etc/motd, and write utmp entries through a third-party library) and doesn't accept any unauthenticated traffic from anyone. To me, security is about these kinds of principles which exclude whole classes of attacks — not about doing something complex and then patching up the security of each part piecemeal.

[kmcallister]

Also, of course, a "same machine" policy is not good enough for people using Mosh to connect to multi-user servers. There's a server at MIT which has 153 Mosh connections from 35 different users.

I know that it's so passé to use operating systems as operating systems, and we should run every program in its own virtual machine, but there are a few of us who haven't switched over to this new way of thinking yet. ;)

[phmarek]

Well, then have mosh tell the filename to the new session-initator, or whatever else you like ...

But, as I already said - with the ptrace syscalls it's possible to inject a new session key and counter value without any server modifications (although a few would certainly help, like the exported symbols). That should be fairly "same machine" only, right?

[phmarek]

I've got another nice use-case for reconnecting to an existing server -- stateless clients.

Imagine that you're running some process in a mosh-server, eg. tail -f. Then, in an CGI script, you connect to the mosh server, get a screen dump, convert xterm control sequences to HTML tags with color attributes, and dump that to the client.

Voila - connecting to stateful processes from stateless ones.

Could you please at least add an exported symbol for the crypto and counter state?

Oh, and I think I misunderstood you - "same machine" + "same user" with a unix socket, mode 0700, in the users' $HOME (or /tmp, like X11 does).

[ayust]

Could the request for a command to kill existing mosh servers (or "mosh servers other than the current one") get split out to a separate issue, or get implemented within the scope of this issue, rather than having it get bogged down by the discussion of the much more complex reconnection concept?

[andersk]

ayust: That’s spelled pkill mosh-server or killall mosh-server.

[ayust]

@andersk No, that kills all mosh-server processes, including ones you might currently be using.
The request is for a command that will clean up disconnected servers.

[tedder]

I'm glad this is still open. Reconnection without using screen is a strong desire for me. Until then it's a nice hobbyist item/proof of concept.

[deutrino]

Workaround installed to my root crontab on a Debian-descended Linux server:

for tty in `w -sf | grep -P '\d\ddays mosh-server' | cut -c 10-15`; do kill -9 `ps -o pid= -t $tty`; done

This kills all mosh-server sessions which are listed as "idle" by 'w' for more than 9 days. I am mildly troubled that I have to do a 'kill -9' rather than a 'kill -15', but there it is.

[heyakyra]

@gordon-morehouse THANK YOU!

[deutrino]

My above workaround doesn't work on my new Linux 3.2 VPS. I haven't come up with one yet to replace it, but I have an open ServerFault question and will post the new method I devise here when I get it figured out.

http://serverfault.com/questions/555467/wrangling-sed-to-find-and-reformat-login-date-or-how-to-kill-idle-mosh-server

[deutrino]

Transplanting a potentially useful comment from @andersk in #426:

FYI, if you use screen inside mosh, then connecting and running screen with a single command:

mosh HOST -- screen -rd

will make sure that reattaching the screen will cause other screen clients to exit, taking their surrounding mosh-server with them.

[tgies]

Here is a variant of @gordon-morehouse's trick that should just kill all detached mosh servers, relying upon the principle that the FROM column in the output of w reads mosh rather than the IP address of the attached client for any detached server (I believe this to be generally the case but I haven't necessarily done extensive checking):

for tty in `w -hs | awk '$3 == "mosh" {print $2}'`; do pkill -9 -t $tty; done

If your w output is missing the FROM column, add -f.

[jaeyeom]

Cool. Thank you very much. I added this line in the crontab. (kill any mosh session that is idle longer than 9 days) Hope it'll work.

@daily for tty in `w -hs | grep -P '\d\ddays' | awk '$3 == "mosh" {print $2}'`; do pkill -9 -t $tty; done

[phmarek]

No, it doesn't have these - but there are other ways.... http://serverfault.com/questions/11317/uptime-of-a-windows-machine

And I'm fairly sure there's some easy way to get the boot timestamp - take the start time of services.exe as a last resort.

[infinity0]

Hi, having mosh-server exit automatically after a timeout is quite important for security reasons. As someone running a server, I don't want to leave holes open if users forgot to close their mosh-client, then they get their laptop stolen or something.

From a more abstract point of view, In cryptographic protocols it's always prudent to set a finite lifetime for the validity of keys (expiry), to provide some guarantee on freshness. Refreshing the expiry by actively using the key is also fine in many cases.

[andersk]

Trying to spin this as a security issue makes for a very weak argument. One could equally well argue, for example, that if idle sessions are discarded, the user may be forced to log in more frequently, thereby exposing more important secrets such as passwords to potential keylogging. You can take almost any feature request and find some way in which it might have inconvenienced an attacker in some specific scenario with some probability—but security is not computed as the sum over all such inconveniences. In both your scenario and mine, the real security violations have occurred elsewhere in the chain. It cannot be Mosh’s place to determine whether a laptop is stolen (anyone who steals laptops would be able to do so within any timeout we might reasonably set); that’s the job of your screen locker and your full disk encryption software.

Personally I think it is fine to discard sessions that are truly abandoned because the client is gone, but not to discard sessions that are merely idle (at least not without asking). I recognize that individual preferences may differ here. But the current contract is that Mosh sessions may be resumed after arbitrary periods of inactivity, and some users may be relying on this contract. So an argument for changing it had better be very strong.

[infinity0]

It's not a weak argument; freshness is a fundamental security property in authentication protocols. You adjust your expiry timeout on a per-case basis, if you want to "avoid logging in too often". Defaulting to infinity with no option to reduce it is most definitely a security hole. Perhaps other non-security concerns are also priorities, but not having timeouts is strictly worse, security-wise.

| You can take almost any feature request and find some way in which it might have [in]convenienced an attacker

I'm quite happy to take this challenge and explain why certain security arguments are worse or better than other security arguments. Your reduction can further be reduced down to "You can say anything about anything, so why say anything at all?"

[andersk]

Mosh is not an authentication protocol in that sense. It runs with no special privilege on either side and delegates all authentication to SSH. Anyone who logs in via SSH could spawn a background process that listens for commands on the same ports that Mosh does and lives for as long as Mosh does. This process could be a version of mosh-server recompiled without any timeout that we might add, or could be one of any number of other significantly more evil tools that have been in use by attackers over a decade before Mosh was developed. If it’s inconsistent with your security policy that an SSH login has this power, that can only be addressed by system-level restrictions, not by changes to Mosh.

If you’d like Mosh to have an advisory timeout because you don’t trust your own screen locker (?), that still sounds like a feature request with costs and benefits to be weighed as any feature request would, not a security problem that overrides other concerns.

In case you find this helpful, I’ll note that screen can do this for you: idle 86400 lockscreen or idle 86400 detach or whatever (enter commands with Ctrl-a : or put them in ~/.screenrc).

[ayust]

Personally I think it is fine to discard sessions that are truly abandoned because the client is gone, but not to discard sessions that are merely idle (at least not without asking). I recognize that individual preferences may differ here. But the current contract is that Mosh sessions may be resumed after arbitrary periods of inactivity, and some users may be relying on this contract. So an argument for changing it had better be very strong.

This seems mostly to be an argument for making such a timeout optional, not for avoiding implementing a timeout at all. Perhaps this could be done via making a timeout something that's requested as part of the initial setup by the client:

mosh --idle-timeout=86400 ...

which would request that the mosh-server which it spawns automatically kill itself if no client is attached for at least a day. This would remain backwards compatible (because legacy clients wouldn't have requested that behavior) but would allow users who desire a timeout to have it (and easily automate it via a shell alias).

[infinity0]

Mosh contains an implicit authentication protocol when it derives a key to encrypt/authenticate the UDP packets with (I haven't actually checked this, god I hope it does this). My proposal is that the validity of this key (from the server's point of view) should have a finite default lifetime, like all good interactive cryptographic protocols should do. When this expires, the server then has no reason to stay alive. Sorry if that wasn't clear, my security argument is about the key.

Yes, anyone could be stupid over SSH and run that other command, or rm -rf, but that is no reason for Mosh to do the same thing.

Relying on a screen locker is pushing the problem to another layer, but really the authentication protocol should take care of this automatically. (edit: actually a screen locker would not even be equivalent here, since you could re-activate it after expiry. it would be like an auto-logout-on-inactivity-command).

[cgull]

Oh, fine. I've posted #690, which adds optional envars for idle-network session timeouts. Go look at it.

@infinity0: Mosh relies entirely on SSH for authentication, and transmits a key from server to client over the SSH connection, and never renegotiates it thereafter. Please look at the source before making arguments about what it's doing. The issues of cryptographic key lifetime are somewhat distinct from those of network idle timeouts and user activity timeouts. Note that if mosh-server does not hear from a client, it stops sending packets to the last-heard-address in a relatively short period of time.

[infinity0]

Great, thanks! I've added some comments. Yes, I agree they are distinct issues, but not sure the best way to express this in the code. Technically, an attacker can withhold the client's packets, then release them slowly to maintain the keepalive. Protecting against that would be more complex and I haven't studied Mosh in more detail to suggest anything concrete at this stage. But I suppose there are other more pressing things to do at this time.

[cgull]

I am probably overthinking this but there's a bunch of possible things you might want to terminate and/or rekey for, from a security and/or admin perspective:

• Traditional idle session timeout (TMOUT). From a security perspective, this is "don't trust a session the user has possibly walked away from on an open screen", from an admin perspective this is "clean up abandoned sessions"
• Network idle timeouts like we're discussing here. I think this is a mostly admin issue similar to the above.
• Key lifetime based on wall-clock time since key negotiation. "How much time has my adversary had to brute-force this session before he can MITM it?"
• Key lifetime based on number of bytes/packets. "How much ciphertext under a given key does my adversary have to analyze? How long can my chosen encryption/authentication run before weakening?" (2**48 messages in OCB's case, so we are told.) This one has the issue that the adversary may see a different set of packets than the client or server do.
• Session lifetime, from login/session start. "How long do we trust this user, after he first authenticated with us?"

[andersk]

Traditional idle session timeout (TMOUT). From a security perspective, this is "don't trust a session the user has possibly walked away from on an open screen",

No, this is “the user doesn’t trust themself not to walk a way from an open screen”, and the obvious solution is for the user to use a screensaver. It is evident that some people want this separate timeout, but I remain confounded by the alleged security relevance.

Key lifetime based on wall-clock time since key negotiation.

All the computers on the planet working together would brute-force a single AES-128 key in about a hundred billion years, and we will be looking for new cryptography long before this number ever threatens to become relevantly smaller.

Key lifetime based on number of bytes/packets.

Since 1.1.2b, we already kill the session after encrypting 2**47 blocks (b4ef664).

Session lifetime, from login/session start. "How long do we trust this user, after he first authenticated with us?"

Again, “how long does the user trust themselves?”. We have no influence over the lifetime of the system’s trust in the user.

[phmarek]

Well, as I'm already active in this issue, here are my €0,02.

In my experience it's really nice to get back to a mosh session after it having been idle for 3 weeks - the last workflow is still visible, it's easier to remember where it was left off, etc.
But having a timeout that's long enough for that case won't really help with the other point raised here - conserving server resources; if my laptop crashes etc., then I typically will establish new mosh sessions within a few minutes, get the message about old sessions, kill them, and continue.

The points about security don't really matter, IMO - mosh won't be used to stream gigabytes of data (like rsync over ssh does), so deriving the key by accumulating enough encrypted data means that all other things using that algorithm should have been broken long ago, I guess.
After quite some use my iptables rule for mosh says 27K packets, 2.8MB of data - that doesn't sound a to attack the underlying encryption protocol...

[infinity0]

@cgull Rekeying long-lived sessions is indeed a good idea, and modern cryptographic protocols like TextSecure's do it. Generally one proves freshness of a remote value by securely tying it to a local event - e.g. doing a key exchange with the local part derived from a random value that can't be guessed. So yes one would protect against the attack I mentioned by continually refreshing the key.

Reducing lifetime of keys is good, independently of concerns such as amount of ciphertext generated or time you think it takes for brute-force (which should, as others point out, be physically impossible assuming good crypto primitives). The main concern here is key theft. Allowing the lifetime of the key to persist indefinitely, as Mosh currently does, would have the same security effect as removing the passphrase protection on the user's SSH private key. If it gets stolen, the attacker gets access without requiring "something the user knows", they only need "something the user has".

(edit: Yes, SSH keys don't have an expiry either; this is not ideal. But when using an SSH key, the SSH protocol at least gives the server the confidence that the key was actually used recently, as opposed to an attacker that is doing a withhold-and-release attack.)

[cgull]

I was overthinking generally there, beyond the bounds of what Mosh is today, and beyond the bounds of what it will ever be. I'm trying to catalogue the knobs that might generally be interesting to have with an interactive session protocol. Since mosh-server is an easily-replaced unprivileged executable, it can't enforce any limits, but implementations could still be useful.

andersk: Remote logins are frequently between different trust domains, and the owner of a server has no control over a user's screen saver. I'm still a little dubious about the security value of session idle timeouts, but it's not obviously irrelevant, not at all. Session lifetime limits have clear value in situations where you may cease to trust a user (this describes any large organization).

And I shouldn't have said "brute force", I should have said "cryptanalyze". Your faith in AES and its 128-bit key length is correct, but there are sometimes other keys to the kingdom. WEP is a fine example of how things can go wrong.

@infinity0: I'm no crypto expert, but I think I agree with your position, especially when you consider applying PFS over the rekeying. But Mosh is deliberately a very simple, relatively easy to review application of cryptography, and extending the crypto implementation in this way would be a major change in philosophy for Mosh.

[andersk]

I really can’t tell if this point is getting through: Mosh cannot help you revoke trust across different trust domains. If you cease to trust a user, it’s your responsibility actively kill all the user’s processes within your trust domain, not leave them around and trust them to politely kill themselves after a while—you don’t trust them, remember? This is why, as I said, the only thing Mosh is potentially capable of doing here is offering to protect the user from their own future self (whether it’s the future self that fails to enable a screensaver, or the future self that fails to revoke their own access when politely requested(?!) by their boss, or the future self that gets possessed by the Dark Lord in a ritual that lasts precisely 72 hours, or whatever).

If the mosh-server binary, or even expert cryptanalysts, could predict the pace of AES cryptanalysis and determine it to potentially complete within human-relevant time scales, the world would already be scrambling to replace AES (just as it has been scrambling to replace RC4).

And I want to emphasize again that the attack with > 2**48 messages is not even just a probably-impractical attack, it’s impossible, because we have already addressed it specifically in the code. (I would not want to be reading through this subthread as a Mosh user for the first time and finding it end with sketchy assurances to the effect of “it’s fine, we think nobody would ever send that much data”.)

[infinity0]

The model you outlined above equates the user with the cryptographic key used to provide access for that user. Or in other words, it assumes that all uses of the key, are by the user that is trusted by the system. Our discussion about key lifetime is to say, no you shouldn't assume that (key <-> user) holds for an infinite amount of time. I can understand if Mosh does not feel the need to solve this with high priority right now, but the concepts being discussed should be clearly expressed at least.

[deutrino]

Hopefully it's useful to point out that beyond security, etc., there are other reasons to do this.

I have a Raspberry Pi which receives frequent logins and disconnects and the like from users on laptops, phones, tablets - clients that tend to reboot or otherwise disappear.

Eventually the memory used by the various mosh-server processes becomes a problem.

[BirkhoffLee]

Any updates on this?

[chrisjohnson]

Also curious. Are there at least workarounds that directly address the original problem without bikeshedding on a solution for another 7 years?

[keithw]

Well, we have the "garbage collection" part now. See the mosh-server(1) man page and the entries on the MOSH_SERVER_NETWORK_TMOUT and MOSH_SERVER_SIGNAL_TMOUT environment variables.

For "reconnection," I think best option for users is just to also use screen/tmux.

[chrisjohnson]

I do use tmux, but my problem is that mosh-server sticks around and I've only opened certain number of ports via firewall. I'll see if the timeout helps with that problem :)

[sumikum7]

I have this issue with tmux/iterm/mosh setup (I use mosh on iterm to connect to a remote server via mosh).
The thing is connection resumption is all fine with iterm quit/force quit but goes for a toss when the MacOS reboots as when it comes back again and I try opening iterm I see that

1. All the sessions(tmux on remote) are already attached(as shown in tmux ls) to their clients 👎
2. I have to now manually enter my remote login password again for all the previous tabs 👎
3. Possibly because of the above two, I am not able to reconnect and it has to open a new tmux session

Then I open a new tmux session to remote and then detach the previous (one by one) and keep attaching them to my iterm tabs as before!

I think this could fix the issue(and they happen fine when iterm is quite/closed abruptly) when iterm hosting OS(macOS) goes for a reboot.

• The best fix for this would be if the remote tmux sessions( on mosh connection) can detect that there is no client.
• Also, I don't have to put in my remote login password again to connect when MacOS reboots.