Transit: protocol improvements

[piegamesde]

A prototype implementation can be found here.

Also note that stun.magic-wormhole.io doesn't exist yet ^^

[piegamesde]

I added support for other transport protocols than plain TCP, thus this PR now supersedes #10. Discussion on the hints format can be found there.

[meejah]

proabably worth adding a point about Tor onion services, since that got deleted in a paragraph above. Perhaps:

• Tor Onion services: one side could run an Onion service, allowing a "direct" Tor connection without using any relay

[piegamesde]

Oh, did I delete too much somewhere? Everything that the Python implementation does should be covered by the spec.

[piegamesde]

Thank you for mentioning the framing, this is a good point: we only require a continuous byte stream (as TCP gives us) because we build our own framing on top. With WebSockets, this seems wasteful. Should I describe a variant that removes the framing for protocols that already bring their own? This would amount to saving the \n in the handshake and then the four byte length prefix per message, which is negligible if you only care about saving bytes.

[piegamesde]

By the way we need infrastructure access to set this up (CC @warner). In the meantime, I host a public instance at stun.piegames.de.

[piegamesde]

I feel a bit weird for bumping the relay version although we technically only change the message formatting and not the way the protocol actually works. I just had an idea for an alternative implementation: the abilities are already serialized as objects, i.e. they may have complex structured values. So instead of having a relay-v2 ability, we could send "relay-v1": { "non-tcp-support": true } or something. The hints field would then become a bit of a misnomer and outdated, but I think that's kind of okay. (An open question would be whether the TCP hints are announced via the old hints field, which is inconsistent, or with the urls, which would make the hints redundant and obsolete.)

[piegamesde]

I implemented the idea described in my last comment, and it turned out to be quite an improvement IMO. Thus, the relay-v2 ability is now gone again, replaced with a url_hints flag on the relay-v1 hint. I took the occasion to also move the abilities description from the file transfer protocol to the relay protocol.

[piegamesde]

🔔 I think this is good to go, so let's start a Final Comment Period! 🔔

I expect all commits except the last one to be rather uncontroversial. If this turns out to be true and the first part does not get any complaints, I'll split this up into two PRs to get that half merged early.

[meejah]

Is the intent of "version A" versus "version B" to decide in this PR on one and only use that, or something else?
I also think a complete JSON example with e.g. two different relays and multiple protocols would be good.

[piegamesde]

Is the intent of "version A" versus "version B" to decide in this PR on one and only use that, or something else?

No. The thing is, that due to how our abstractions are layered, that transit cannot specify how the required messages should be encoded. This is part of the application protocol on top. Usually, there is a "canonical" encoding that everybody follows, like for the hints. For the abilities however, it has some (minor) drawbacks that are improved in Version B. That one is not used anywhere at the moment, except by my port forwarding and file transfer v2 experiments.

That being said, I'll remove Version B and mark Version A as "canonical". This will make it easier to understand. It does not change any semantics, since the upper layer protocol still has the last word on how things should be done.

I also think a complete JSON example with e.g. two different relays and multiple protocols would be good.

Will do.

[meejah]

I will do a more in-depth review tomorrow and/or weekend, but a couple quick thoughts:

1. I do think adding a "name" might be useful while we're changing the hints. The way I think of this is that there are some number of relays (currently "1") and they each have some number of ways one can connect to them. (One use-case here being inter-op -- consider a Web client that only cares about wss hints and a Python client that only cares about tcp hints -- if they used the same relay that supported both, they can relay to each other). Another use-case is simply something nice to display to users.
2. the "urls" stuff feels like it wants to just be another "type"; maybe once there's a bigger example this will be more clear, but a flag like that makes me wonder why there isn't just a type: "url" in the list (instead of a flag and a separate list).

[meejah]

looking good!
see also #16 (comment)

[piegamesde]

All comments are resolved, maybe we all read through it once or twice to catch even the last typo and then it should be good to go 🎉

[meejah]

awesome (and sorry this took so long to get merged)