QUIC as Alternative Carrier for the Erlang Distribution

[mickel8]

Is your feature request related to a problem? Please describe.
This feature is not related to any problem.
QUIC is a new transport protocol standardized in RFC 9000 (and RFC 8999, 9001, 9002) and it provides interesting set of features.
The idea is to provide alternative carrier for Erlang Distribution that will use QUIC instead of TCP/IP.
User should be able to choose which carrier it is willing to use with default pointing to TCP/IP.

Motivation:

• number of applications does not need to exchange data reliably and they want to bypass delay resulting from TCP retransmissions. Using QUIC we could send control data in a reliable way using STREAM frames but allow user to choose how to send its application data -- either via STREAM or DATAGRAM frames (DATAGRAM frames are not standardized yet)
• Distributed Erlang could benefit from faster QUIC handshake comparing to TCP+TLS. The problem here is that QUIC does not allow for unencrypted communication that seems to be default in Erlang Cluster

Describe the solution you'd like
The idea is to provide alternative carrier for Erlang Distribution that will use QUIC instead of TCP/IP.
User should be able to choose which carrier it is willing to use with default pointing to TCP/IP.

Describe alternatives you've considered
I am curious what you think

Additional context
none

[vans163]

I dont see the usecase of this for erlang cluster, but I would really like QUIC as an alternative protocol similar to the http module, so you are able to connect or accept connections using it.

One big problem QUIC solves is that its "stateless" so if a user is tethering and switches mobile towers or between wifi->4g, QUIC will stay connected, while TCP will drop the connection and force the user to reconnect leading to unpleasant experience.

[essen]

The number one advantage of QUIC over TCP for the Erlang distribution is streams. QUIC data is guaranteed to be ordered on a stream level. In the context of the Erlang distribution, a stream could map to sender_pid->dest_pid, or to sender_pid (to avoid having too many streams), because Erlang only guarantees ordering of messages between two processes. The control stream could be used for all operations not directly related to processes (such as node monitoring).

What this effectively would solve is head of line blocking. Current Erlang versions do have fragmentation to avoid this issue, so whether QUIC would improve anything further remains to be seen, pending experiments. It is already possible to build custom Erlang drivers and there's at least one QUIC NIF, so it is possible today to do one such experiment. No doubt a proper implementation would perform better as the current interface for distribution drivers doesn't make easy using a concurrent protocol (you have to disable the atom cache etc. since it expects total ordering).

Of course the number two advantage is the built-in TLS and security by default but that could be achieved over TCP as well.

[mickel8]

Unreliable data exchange between nodes could be beneficial in interactive communication scenarios.
Consider video conference room split across multiple nodes.

[essen]

Sure but you are talking about a use of the distribution protocol that does not currently exist. The distribution protocol today requires reliable and ordered data transport (for the most part, as I mentioned). There is no unreliable or unordered channel. So before you get to this point you will need to expand on the protocol to allow for this type of communication. It will most likely be easier to spin up your own connection (optionally reusing the same configuration as you would for TLS/QUIC dist) for quite some time.

[okeuday]

You may want to consider using gen_sctp in addition to QUIC support or instead of QUIC support, for usage as an alternative carrier for the Erlang distribution protocol (ideally an example would be present here). While QUIC functionality is a subset of SCTP functionality, SCTP should have more work done in the kernel than QUIC when comparing the source code executed. SCTP wouldn't be using encryption which would help make it the more efficient option for LAN use (normal distributed Erlang use would be limited to a LAN, so encryption only adds latency unless there are virtual host security problems to be concerned about).

[mickel8]

Good point. SCTP is such a great protocol in my opinion. The only problem with is that it operates directly on IP so a lot of middleboxes does not recognize its packets.

[okeuday]

@mickel8 For dedicated hosts on a LAN using a switch there is no problem due to the switch using the data link layer. That is the ideal situation to avoid extra complexity for fault-tolerance concerns. However, virtualized hosts are a major concern too and related network functionality may block SCTP use at the transport layer (normal routers would only care about the network layer (IP) but firewalls would cause problems if they lack support for SCTP).

[essen]

Not using encryption is a negative point for an increasing number of customers, even over LAN.

[voluntas]

I'm developing a commercial real-time audio and video distribution server using Erlang, and I don't see the need to apply QUIC to Erlang node communication.

We don't think it's necessary to apply QUIC to Erlang's node communication, as we believe that cluster communication is basically supposed to be done on a private network. If you're planning to use a public network, it's better to implement it on the Erlang VM itself, so that it can be optimized for the delivery protocol. We're going to implement it ourselves.

In fact, we may even adopt SCTP over DTLS over UDP, which has a proven track record in WebRTC.

[lin72h]

Not directly related for the topic, Emqx has been done some work on nif implementation of msquic, and it will be intergrated into their next release of commercial product

[mickel8]

After some time I implemented this on my own. You can check it out here https://github.com/mickel8/quic_dist

[heri16]

Would love to see some benchmarks