Multiplexing more sessions into same UDP port

[lasseleegaard]

Currently mosh uses a port range to support multiple sessions to the same host. In the corporate world with many firewalls and many restrictions, opening ranges a thousand ports wide is always a hassle if not impossible. Looking at it with a security mindset it does not look much better.

So - the feature request is:
Create option/change default behavior to allow mosh to use one single UDP port on the server side to multiplex more sessions into - just like we are used to with SSH and many other protocols.

The use of UDP/22 could be considered and/or the registration of an new low port number at IANA.
Alternatively - just make it configurable.
One of the advantages of using port 22 is that it is already used for SSH and interactice shells and convincing the Mordacs of this world to allow UDP in addition to TCP would be easier.

[keithw]

I think we could probably do this. The best idea I've had for how it would work is:

1. New clients pass a "-p" flag to the server to indicate they are multiplex-capable.

2. The server continues to open a local port (like 60001). But it also reads a config file (like /etc/mosh.conf) telling it to instruct incoming clients to send on, e.g., UDP port 22. It prints a MOSH CONNECT line like "MOSH CONNECT 60001 22 [key]".

2b) If there is a /etc/mosh.conf specifying use of a multiplexing daemon but no "-p" flag, the server prints an error message telling the user to upgrade their client.

3. The client uses 22 as its real port number. But it also prepends a 16-bit network-order integer containing the 60001 to every datagram.

4. "moshd" receives these datagrams on port 22 and resends them to the indicated local port ("indicated" at the beginning of each client->server datagram).

Characteristics of the design:

1. Clients and servers both have to be upgraded to use multiplex, but existing clients will still work with new servers that don't require a multiplex.

2. When an existing client tries to connect to a multiplex-requiring server, it will get a clear error message.

3. The server doesn't have to communicate with "moshd", doesn't have to reserve any resources with it, and doesn't have to worry about something lingering when it shuts down.

4. Remote hosts will gain the ability to send UDP packets to ANY port on the local machine.

5. The "multiplex" port number field becomes an added header on client->server datagrams, but we don't otherwise modify the SSP protocol.

[keithw]

If you can think of any idea for how this could work with existing clients (by only modifying the server), I would love that. You could imagine a moshd that would take incoming packets (with no port number prepended) and somehow infer which subset of mosh-servers to send them to (maybe by tracking the sequence numbers, or maybe by letting the mosh-server request all incoming packets from a particular IP address that it's "attached" to?). It's ok to send the packets to multiple mosh-servers because the ones where it doesn't decrypt will just ignore it.

But this seems like it may be a flakier design overall than explicitly indicating the port number.

[nicowilliams]

@keithw:

I assume the port 22 UDP proxy would add client source address/port info to the datagram before forwarding to the requested protocol.

The thing about clients getting to send UDP to any port on the server is... not cool. Options include: a) the server tells the proxy what port it has open then the proxy allows datagrams only to those ports, b) define a UDP port range for this? but this gets us back to session number limits, c) ?? I prefer (a). You'll want the server to tell the proxy what port it will use in a way that other unprivileged local processes can't.

[nicowilliams]

Also (though you already knew this, it's worth restating): this is probably the single most important feature w.r.t. getting enterprises that have SSH gateways to setup Mosh gateways. Godspeed.

[keithw]

Yeah, we may also want to wrap the whole thing in another encryption key (which the moshd would have), to satisfy the people who hate that we send sequence numbers and port numbers in the clear. (Even though SSH, TLS and DTLS send sequence numbers in the clear too.)

[nicowilliams]

Seems like a waste to add that additional encryption when the ciphertext will be going only over IPC to... processes spawned by the sender. What does matter is that users not be able to extract that info out of mosh servers, and that there not be a DoS as a result of re-writing the port number in the datagram sent by the client (but the latter was already a requirement).

The protocol by which a server gets spawned and it then tells the proxy what UDP port it opened, that is critical.

[calmh]

While this would be lovely and is for sure necessary, this:

4) Remote hosts will gain the ability to send UDP packets to ANY port on the local machine.

is not a desirable feature...

[keithw]

Sorry, I should be clear that this "additional encryption" would be going from the client to moshd, and would encrypt the "real" port number and sequence number. This is to satisfy the people who want there to be some ambiguity between different Mosh sessions to the same server.

We probably will need to find a way to stop (4) as well (remote hosts gain the ability to send UDP packets to ANY port).

[calmh]

@keithw How would there be any ambiguity, given there'll be a different remote IP and port number for each session?

(Not that I have anything against encrypting the port and/or sequence numbers if someone really wants to, I just don't see what extra security it buys anyone... However point number four above would be a very serious flaw if implemented.)

[nicowilliams]

@keithw: well, you're already not getting privacy protection for the target port number (you're not using ESP). Sure, you could get a slight measure of traffic analysis protection from encrypting this in the packets to the proxy, but it's not much when you consider that the replies will come back from an identifiable port number on the server. To me this would be unnecessary complexity since now you'd have to have two sets of session keys: one for the client<->proxy and one for the client<->server.

[keithw]

You don't have to tell me! Participate in the discussion on mosh-devel if you want to have an impact. People like Jacob Appelbaum and Seth Schoen want this, despite the fact that the privacy improvements will be only statistical (and will depend on the presence or absence of credible chaff).

[nicowilliams]

Guys, you reject crypto agility, and generic composition of unencumbered AEAD cipher modes on complexity grounds but would accept a dubious traffic analysis protection feature? Come now, be consistent :)

[nvx]

Rather than relaying the packets to another UDP port on the local machine, why not open a UNIX socket in a specific directory (specified by the mosh.conf file) such as /var/run/moshd/60001.sock, then 60001 refers to the UNIX socket's filename, not a udp port.

Would potentially break when servers are running on non-*NIX-like platforms, but you could easily allow falling back to local udp sockets in such an instance (and it's not likely to be a major use case of mosh having the server run on such platforms anyway). Another option is to have mosh.conf specify a whitelisted udp port range and only those ports can be used, the mosh-client respecting the port range too.

Additional encryption seems rather silly, I'd compare the security to what ssh currently offers, if ssh doesn't offer it, there's no real gain to offering it with mosh unless it can be done for free. Encrypting already encrypted data doubles your encryption overhead, and is hardly free, not to mention any possible key negotiations required...

[grawity]

A mosh.conf sounds good, actually, but it would have to be per-user (residing in ~/.config/mosh/ ideally) since I've seen at least two cases where different users were assigned different ports.

[nicowilliams]

@nvx: +1

[zedinosaur]

Is there still interest in implementing this via the approach described by keithw?

[lasseleegaard]

Yes please.

On Tue, Dec 31, 2013 at 9:38 AM, zedinosaur notifications@github.comwrote:

Is there still interest in implementing this via the approach described by
keithw?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/295#issuecomment-31387822
.

[KwadroNaut]

I think it would still be really nice.

[eMBee]

instead of the client referencing the port it could reference a session id. the id then maps to the actual port in use. that way only mosh ports can be accessed.

greetings, eMBee.

[zedinosaur]

Making it a session ID will then require some sort of negotiation between "moshd" and the mosh-server. If we can encode the actual port number it keeps things simpler.

I would actually modify keithw's proposal slightly and instead of just adding a static additional 2 byte value it I would abuse the second highest bit in the seq number to add a flag for whether a new, unencrypted header is present. Then after the seq number add a byte for the size of the unencrypted header. Then the unencrypted header would just be a proto and could contain the port number. These makes the overhead 5 bytes instead of 2 but makes things must easier to extend in the future.

[nvx]

Assumptions:

1. If you are running moshd, mosh, and thus, moshd would be installed system wide
2. As moshd is installed system wide, a global config file can be referenced by mosh and moshd for config options (such as internal and external port, which may or may not be the same depending on nat etc - although ideally would coincide)
3. A directory with appropriate permissions could be created (/var/run/mosh ?) which would contain UNIX sockets instead of UDP sockets for use by mosh <-> moshd communications - the location of which can be discovered by the config file. An additional socket can be opened by moshd to allow mosh to say "Hi I just opened a socket at (which is required to be in the /var/run/mosh directory), here's the session id".
4. A session will not roam between moshd and non-moshd - so there is no need to have both a UDP and UNIX socket open at once - this may be of importance if you expect a LOT of mosh sessions (busy shell server) and/or wish to restrict the ability to listen on tcp/udp sockets by users.

[zedinosaur]

What would be the appropriate permissions on that directory? World-writeable? And to make those sockets work they'd either need to be world readable/writeable or moshd would need to run as root. I think getting the permissions right here would be tricky but keeping everything on UDP makes things easier permissions-wise.

[nvx]

Socket permissions would have to be fairly permissive, for example:

srwxrwxrwx 1 mysql wheel 0 Jan 1 19:04 /tmp/mysql.sock=

The advantage of using UNIX sockets is it immediately solves the problem of using moshd to bypass firewall rules to talk to arbitrary UDP sockets.

[zedinosaur]

OK thinking about it more this I think the socket based approach could work well. When doing the ssh handshake the mosh-server could generate a random 32-bit session id and open a socket based on that id (eg /tmp/mosh-). Then the client would embed the session ID in the packets as I described above and moshd would use that to figure out which socket to send the packet to and forward it there. Hopefully we could use SCM_CREDENTIALS to make sure we're only getting messages from who we expect.

[earlchew]

zedinosaur, are you already working on bringing this feature to life ?

I've been thinking about this exact issue, and have just stumbled across this thread. I've got one hole punched in my firewall for mosh, and don't really care to punch any more holes.

I was thinking about the simplest way to get multiple mosh connections without having to perturb the mosh codebase too much.

I thought this might be possible (at least for prototyping) using the "--client" and "--server" options on the mosh client front end. I think this would allow the current non-multiplexing client and server to remain unchanged. Normally a mosh-client connects directly to a mosh-server. In the multiplexing case:

mosh-client --- mux-client-endpoint <-----------------> mux-server-endpoint --- mosh-server

Effectively, mosh --client="mosh-client-mux" --server="mosh-server-mux" could set this up. For example, the program mosh-client-mux program could configure and run the mosh-client and mux-client-endpoint programs on the client machine. Similarly on the server side.

Once prototyped, the mosh-client-mux and mux-client-endpoint could be folded into the mosh-client, and mosh-server-mux could be folded into the mosh-server. The mux-server-endpoint would have to remain as a separate piece of functionality (effectively moshd).

While this scheme nicely allows the mosh-server to remain UDP based and run with few if any alterations, the problem is that the mux layer needs to know the UDP port on which the mosh-server is listening.

[earlchew]

If introducing a new transport layer to mosh-client can be considered, then the suggestion to use a Unix domain socket is a good one.

Using this approach, I would think about having moshd (the server endpoint) accept UDP packets from external mosh-clients, and also listen and accept Unix socket connections from local mosh-servers.

The name of the Unix socket that moshd is listening would be known to mosh-server or published somewhere.

When instructed to use the multiplexing mode, the mosh-server connects as a Unix socket client to moshd. All connection data is subsequently passed to and fro on this connection.

Creating the connections in this way avoids the need to have a (potentially world writable) directory filled with connections, pipes and sockets that come and go.

[hirvinen]

Hi,

A few thoughts :

1. In some cases where the administration doesn't mind leaving a thousand ports open, it is valuable for unpriviledged users to be able to install mosh for themselves, so that may well be worth retaining.
2. In systems where there is only a local OS firewall, mosh could be granted the right to open ports in a root-configurable range via sudo (I'm thinking of doing this for my own server) but hooking into all possible firewall management systems would probably be problematic(at least a lot of work) or not very user-friendly.
3. When user runs mosh-server, it checks for a running moshd and if it doesn't exist, everything proceeds as now. If it does exist:
   * mosh-server asks moshd for a port/socket, a sessionID and (optionally) moshd's public key, the last two being passed on to the client.
   * When moshd public key is present, mosh-client sends crypt(sessionID+crypt(payload, sessionKey), moshdKey).
   * When moshd receives something on the external port it is listening on, it tries to decrypt. If successful, and the user who got that sessionID still has mosh-server running on the same port, the packet will be forwarded there.
   * moshd routes return packets based on the mosh-server and its sessionID, and encrypts them with its own private key.

This should be doable with relatively little modification to either mosh-server and mosh-client with relatively thin wrappers if desired.

[earlchew]

I've collected my thoughts together in a more concrete manner at : https://github.com/earlchew/mosh/blob/issues/295/dev/issue-295.md

[nvx]

I can't t think of a better way to do it than how you've described there @earlchew - that has my vote over all the other suggestions!

Should hopefully solve the NAT problem too.

[earlchew]

I've added a candidate design diagram at : https://github.com/earlchew/mosh/blob/issues/295/dev/issue-295.md

[notorand-it]

Is there any news about this very feature?
The linked "diagram" by earlchew seems to be stale and actually missing the diagram itlself.

[cgull]

@earlchew's proposal is the formal, correct approach to solving this problem. But I have an idea for a pragmatic, simple solution to this problem: Take @earlchew's proposal, but strip out the session ID-- moshd and mosh-server just wrap packets in the existing format with a header indicating the remote IP/port. Instead, moshd keeps a dictionary of the remote hosts sending incoming packets, and remembers which remote host any given mosh-server responds to. On receipt of a packet from a previously unknown remote host, simply send it to all connected mosh-servers. Since a mosh-server simply drops packets that aren't for its session, either one or zero mosh-servers will respond to the packet. moshd can obviously fairly quickly figure out which mosh-server is associated with that remote host.

This scheme has one big win and one big lose:

• Win: No change to mosh-client is required, all of this can be managed server-side. Big compatibility win.
• (Smaller win: Somewhat easier implementation than earlchew's proposal.)
• Lose: This broadcasts unknown packets to all mosh-servers, which won't scale and has obvious DDoS related implications. It'd probably be OK for a host with ≤10 sessions, but things would get ugly at 100 sessions. I suspect very few servers would need to support >10 sessions, though.
• Lose: mosh-servers get to observe when clients other than their own roam to a new network. (mosh-server currently exposes similar information in utmp, though.)

[nvx]

Clever, I like it.

The last lose isn't really a loss as the information is already exposed anyway.

For the DDoS implication isn't really that bad anyway - in most cases the moshd would be running on the same host as the mosh-servers anyway, so no bandwidth exhaustion, just some potentially wasted CPU time (other hosts would likely have their own dedicated UDP ports forwarded). Or if there was a dedicated moshd for multiple hosts, it'd likely be on the same LAN which would have much more bandwidth than the WAN link.

Additionally, moshd could be configured to rate-limit roaming attempts, and the moshd UDP port would likely be unknown to an attacker (a firewall could be configured to simply blackhole probes to unknown UDP ports, which would be the same behaviour if a packet was sent to moshd and was unknown by all mosh-clients) unless said attacker was able to successfully authenticate to the ssh server to start a mosh-server of their own to retrieve the configured UDP port for moshd which would further limit such an attack (and if an attacker can ssh to your box, chances are there's better ways they could launch a DoS attack against your things).

Also worth noting, if you're expecting to support hundreds of connections, chances are you're going to have a public IP you can throw on the box anyway so you wouldn't be using moshd anyway. Usecase for moshd I see as more for smaller use.

Big win for client compatibility rocks though.

[cgull]

There is at least one more problem with my scheme that @earlchew's proposal avoids: an untrusted local user can connect to moshd and reply to all packets. This doesn't have to be a DoS for other local users, moshd can just forward a given remote to all repliers, but the bad actor does effectively get tcpdump on one direction of everyone else's session. The classic solution here is to restrict access to the moshd socket to setuid mosh-server binaries, but that adds a new attack surface to mosh-server, of course.

[cgull]

Also, since my proxy scheme makes no attempt at authentication, it could be used as a semi-generic UDP proxy. That would be bad in some situations, if you have users that you can't trust.

[notorand-it]

I wish I could give a look to earlchew's "diagram".
Any way to view it?

[earlchew]

The previous method of embedding diagrams seems to have rotted. I've updated the link and it seems to have fixed the issue.

[lenormf]

Any news on that front ?

[guillaumeautran]

Will there be a fix for this at any given time?

[notorand-it]

I don't think this will happen. The application would become a completely different one.
As it's now you don't need an extra server on the destination host.

[achernya]

This feature becomes tractable if we ever have a privileged "moshd" listener to route the user sessions.