document websocket support in the relay server

[vu3rdd]

No description provided.

[meejah]

LGTM

[bryanchriswhite]

I feel like these should be called relay-ws-v1 and relay-wss-v1 but I also feel like we had this conversation before and I don't recall how it ended. Could you remind me what's up with that?

If we did go with the above, would it then make sense to rename relay-v1 to relay-tcp-v1 (in a backwards compatible way w/ deprecation warning)?

Perhaps we can add descriptions of these in the list that follows? Maybe something like "indicates that it can connect via the transit relay using <proto> transport".

[piegamesde]

I know it's kind of tangential to your change, but why would one advertise any of the other three values? What meaning do they have in the protocol/handshake? I think the only relevant question is "can we connect directly or do we need a relay"?

[vu3rdd]

What you are saying makes sense, now that relay specification also includes the connection protocol part (as in tcp:foobar.org:4001 or ws://foobar.org:4002). It wasn't there in the past before the WebSocket changes.

[vu3rdd]

Perhaps those changes should be separate from this change as a "-v2" abilities? That would also help maintain backward compatibility with older relay servers and clients.

[piegamesde]

I'd like to discuss the semantics of having multiple protocols towards one relay server. What is the intended feature negotiation once both sides have exchanged their abilities? Usually, I'd say both sides have to agree on each (i.e. take the intersection of both sets), but with web sockets this needs not to be so: two clients could connect through a relay, one with web sockets and one over TCP. (Corollary: do both sides actually need to use TOR in order to use TOR or is only one side sufficient if additionally a relay is used? I don't know how TOR works internally at all.)

[bryanchriswhite]

@piegamesde I can't speak to tor because I too am not well-read on tor. Additionally, wormhole-william, which we're using for the current project, hasn't implemented tor support yet.

I'm not aware of any reason that the relay wouldn't be able to speak a different protocol on each leg. @meejah implemented the transit relay side of the change so perhaps he has more concrete details here.

The original motivation for us to add a WebSocket as a transport option was to support a web environment. In this context, use of Websocket transport implies use of the relay (as browsers can't listen for incoming connections*). In a scenario between clients which both support TCP and WebSocket transport, the relay is not necessarily a given; thus, I think the following would apply from the transit spec:

To tolerate the inevitable race conditions created by multiple contending sockets, only the Sender gets to decide which one wins: the first one to make it past negotiation. Hopefully this is correlated with the fastest connection pathway. The protocol ignores any socket that is not somewhat affiliated with the matching Transit instance.

[piegamesde]

Okay, this poses a few problems. Let's assume that relays are able to bridge multiple procotols (this should be mentioned in the spec, so that clients can relay on it). Then, at the moment, there is the possibility for a nasty race:

1. Client A has ws:myrelay.com, client B has tcp:myrelay.com:443. Both support both.
2. After the capabilities exchange, both have both URIs. Thus, both connect to both URIs at the same time.
3. A has a better internet connection, and gets a connection to both relay endpoints first. By the current relay handshake, A will now be looped to itself, and the handshake will fail.
4. I don't know what happens for B, but probably the same.

Thus, if a relay has multiple endpoints, clients need to know this in order to deduplicate somehow. Alternatively, we can adapt the relay handshake to handle this case, but I have no straightforward idea for that. A third option would be to mandate that all relay servers always support all endpoints, but this may be troublesome for backwards compatibility reasons (esp. we'd run into the problem again when adding a third endpoint).

[bryanchriswhite]

@piegamesde in a scenario where both clients support multiple transports and the relay is used, I think it makes sense for the client to only connect to the relay with one transport at a time.

Good observation!

[piegamesde]

I don't really know how to move this forward. I also kind of depends on how willing we are to make big breaking changes to the relay protocol. The simplest solution I can see is to add the client side in the relay handshake:

please relay $token for $side\n

That way the relay will know if two connection attempts stem from the same entity and can safely ignore the all except the first one. Note that this is not 100% backwards compatible on the Relay side, but at the moment there is only one and it is in our control so I don't feel that bad about it.

While it solves the problem with the race hazard, it still does not resolve the other problems I see with the current abilities merging: In theory, an application supporting only websockets can communicate with an application supporting only TCP just fine, provided that the relay supports both. But there isn't really a way to tell whether a relay does, so it may just horribly fail.

[meejah]

The relay server already uses please relay X for side Y (although it still works if you use please relay X .. because that's the original protocol).

[meejah]

For Tor (which is not an acronym so they prefer "Tor" over "TOR"): with client-to-server situations, a Tor connection can be thought of as just a normal TCP connection except that the server doesn't know where the original client is on the internet. So, either client can use Tor to connect to the relay server (for example) independently of each other.

An even better way to use Tor would be for one side to set up an Onion service and advertise it to the other. An Onion service is a Tor way to be a server. Clients of such a server have to use Tor. In this setup, the relay server would not be needed but there also won't be any NAT issues because all connections to the Tor network are "outgoing" (including for Onion services). (We don't yet support this mode though).

[meejah]

The current relay implementation can handle either TCP or WebSockets on either side (so can relay WS-to-WS or TCP-to-WS on either leg).

[piegamesde]

Thank you for the input. In that case I suggest the following changes to the protocol:

• Document the please relay X for side Y and what this means for relay servers
• Document that self-loops may happen when not specifying the side
• Keep the abilities set to direct-tcp-v1 and relay-v1 (do we need to bump the version here?). I thought about having more flags and only sending relay URLs to the peer that they'd support. But then I figured that input sanitation will needed to be done anyways, so I don't see the point in doing this. Maybe reserve tor-hidden-v1 for "I am able to spin up a hidden service" or something.

[meejah]

I think all that sounds good @piegamesde .. it might be that we can side-step the "loops" problem: any client new enough to support WebSockets should also support the for side Y part of the handshake. And I think the only case where a loop might happen is when a client is using TCP plus some other kind (right?).

That said, we should probably still document the issue (and that new clients should always use the for side variant).

Edit: we could enforce this by insisting that WebSocket clients can only use the please relay X for side Y handshake...

[piegamesde]

And I think the only case where a loop might happen is when a client is using TCP plus some other kind (right?).

That's the obvious case, but no. It may end up happening whenever a client has two entries with different content to the same server. Yes, I don't expect this to ever happen, but I still think it should be documented that it might happen as it clearly is not impossible.

we could enforce this by insisting that WebSocket clients can only use the please relay X for side Y handshake...

It'd only partially help (see above), but it's a good idea nevertheless.

[bryanchriswhite]

The relay server already uses please relay X for side Y (although it still works if you use please relay X .. because that's the original protocol).

I went looking for this in the spec because I thought i recalled something about "for side ..." being a thing but didn't see it. Was this a change that just didn't get documented?

[meejah]

@bryanchriswhite maybe? That handshake was already in there when I implemented WebSocket support ...

[piegamesde]

It was already there when @vu3rdd implemented transit in the Rust port a few years ago. But I've bisected the original commit with that change, it's from the end of 2016 and it contains no further motivation: magic-wormhole/magic-wormhole@e1546bf

[piegamesde]

Coming back to this again, this time with from the perspective of adding UDT transport (personal experiments) and advertising relay servers in the welcome message. Some random thoughts:

• Clarify the relay handshake as discussed above
• The abilities exchange should be a separate step from the hints exchange, because the hints require some setup based on the peer's abilities. The spec already kind of mentions this, but it should be made more clear. This won't help us right now since there is no way to change the transfer protocol to respect this, but an eventual transfer v2 will have to correct that mistake.
• Relay servers (and thus, relay hints) are advertised with all supported protocols, independently of the other abilities (if the relay supports tor, include its onion address for example). The hints are bound to the server so that only one connection needs to be made per server. Hints can be as simple as an encoded list of URIs (edit: actually they're even URLs), where the schema determines the protocol.
• The tricky bit is to update the current transfer protocol to work with this. I suggest adding a new relay-v2 ability and deprecate the old.
  • Older clients ignore the relay-v2 hint
  • Newer clients must send a relay-v1 hint alongside the v2 that includes the encodeable subset of the information (notably, the server's TCP address if present).
  • Newer clients ignore the v1 hint if a v2 hint is present
  • If abilities are sent separately from hints (sadly not our case right now), and both clients signal a relay-v2 ability, then the v1 compat may be skipped.
• General protocol clarification: clients MUST support relay servers, and they SHOULD send that ability. The only acceptable exception is when a direct connection attempt is being enforced. On the other hand, clients don't need to support any direct connections, relying solely on the relay. Clients that wish to hide their IP address SHOULD NOT use direct-* hints.

If you want and find this proposal acceptable, I can draft a PR containing the spec clarifications.

[meejah]

Relay servers (and thus, relay hints) are advertised with all supported protocols, independently of the other abilities (if the relay supports tor, include its onion address for example). The hints are bound to the server so that only one connection needs to be made per server. Hints can be as simple as an encoded list of URIs (edit: actually they're even URLs), where the schema determines the protocol.

I definitely think it's important to scope multiple addresses + schemes to a single logical "server".

This might need slightly more thinking as to whether URLs are sufficient (certainly they are for many use-cases). Twisted's "endpoint strings" runs into this problem a little, though: "Tor" is a wrapper protocol, so you could use "tor+ws://..." or "tor+tcp://" for example and WebSockets could have multiple transport addresses that may not always map to a URL as simply. Although from a server perspective, the only "Tor only" transports are going to be .onion addresses.

For example, a WebSocket server may be reachable via multiple addresses, but still wants a canonical URL ("ws://example.com/some/path" may be reachable via a 192.168.* address as well as possibly multiple public addresses). Autobahn addresses this by allowing separate transport configuration from URL configuration (although for a the "usual" straightforward case you can skip that and just do a straight URL).

[piegamesde]

First of all, now that I think more of it, there is no need for explicit TOR support: .onion servers don't make any sense (if both sides support TOR anyways, they can connect "directly"), and a server doesn't care whether you tunnel through TOR to reach it or not. (Similarly, a client can tunnel over TOR for any TCP-based protocol at well.)

I did not say to only specify one URL per protocol and server. I was saying that we should group relay URLs by server so that a client only needs to pick one per server.

[meejah]

First of all, now that I think more of it, there is no need for explicit TOR support: .onion servers don't make any sense (if both sides support TOR anyways, they can connect "directly")

I don't think that's true; they'd have to go via a relay server then. Then point of one side running an .onion is so that they can indeed connect "directly" over Tor.

(p.s. it's "Tor" not "TOR").

[meejah]

I did not say to only specify one URL per protocol and server. I was saying that we should group relay URLs by server so that a client only needs to pick one per server.

Yeah, I get that: a list of URLs per server. That may indeed be sufficient -- I was just trying to bring up some of the weirder edge-type cases. But, you could just list ws://192.168.1.1/foo alongside ws://example.com/foo I suppose (for the multi-homed host example above).

[meejah]

...Also, regarding Tor, it may be that relay servers want to offer services in a network-location-hiding way. Really, that's what .onion services are for: network-anonymity for servers. That is, there are at least three kinds of Tor use:

• a client contacts a normal, TCP-using server (websockets or not) over the Tor network. This is completely up to the client.
• two clients wish to connect "directly" via Tor; one has to decide to run an .onion service and the other connects to it.
• a relay operator wishes to hide the network-location of their server and only listens on a .onion address (thus forcing clients to use Tor)

[piegamesde]

At the moment, only the first mentioned use case is actually implemented by clients. Use case two is covered by the tor-tcp-v1 hint, and thus is out of scope of this discussion. Use case three is redundant to the other two in my opinion, but my guess at specifying this would be to add a tor-relay-v1 ability & hints. (Actually, this is almost exactly the same thing as the second, except that the clients first do a relay handshake upon connecting.)

[piegamesde]

Superseded by #16.