Review quinn crate for possible uses

[fhaynes]

https://github.com/djc/quinn is an implementation of QUIC. We should include reviewing it for 0.2 along with @eaglgenes101 work here: https://gist.github.com/eaglgenes101/55466ff21a2e7184e6499211086ea725

[eaglgenes101]

BTW I've since updated the sketch (after some rereadings of the protocol, among other things): https://gist.github.com/eaglgenes101/9a37e7e15817ca76b6d96a5c54cdaf20

[Ralith]

As one of the quinn maintainers, my primary goal is specifically to develop a cutting-edge transport protocol implementation for realtime multiplayer games, so if there are any barriers I'll be happy to help.

It should be noted that QUIC can easily be applied out-of-the-box for any combination of reliable/unreliable ordered/unordered messaging. The partial reliability extension that @eaglgenes101 cites reduces demands on the application for the unreliable-ordered case, which is useful, but no extensions are required for the necessary primitive operations to be available.

[eaglgenes101]

Fourth take. No need for too much extras, just plumbing of what already exists.

https://gist.github.com/eaglgenes101/1a7d7350bebdbdfe0244915921b46d6a

[Ralith]

That seems basically reasonable. A few notes:

• Requiring EXPIRED_STREAM_DATA merits justification. You illustrate how you'd use datagram frames as an optimization for small unreliable messages, but what's the specific requirement for partial reliability within a single stream?
• There's no requirement for ALPN IDs to be ASCII; encoding things as base64 or hex is just wasting space, although there's usually a lot of dead space in the initial packet anyway for security reasons so it's not a big deal. In fact, including any meaningful structure in an ALPN ID at all (as opposed to UUID-style random bytes) is only of benefit to someone looking at logs or inspecting network traffic, or if you want to register with the IANA someday.
• It might be useful to specify the exact reset strategy for unreliable messages transmitted as streams; do you want to abandon them as soon as every segment has been transmitted at least once, or when some higher-level logic has decided that they're obsolete?

[eaglgenes101]

Fifth take, taking the above issues with the previous iteration into consideration:

https://gist.github.com/eaglgenes101/a99438772511ecfc0542f3377796bb9e

[Ralith]

The extension you're proposing has some issues.

if a STREAM_IMPLICIT_RST frame is received after a STREAM frame, a STREAM_IMPLICIT_RST frame is received after one with a different instance ID, or a STREAM frame is received after a STREAM_IMPLICIT_RST frame for a given stream ID, the frame is interpreted as a stream reset

This will lead to inconsistent behavior under packet reordering. You can't rely on receiving frames in the same order they're sent.

The intended interpretation of STREAM_IMPLICIT_RST is that each independent message sharing the ID is associated with an instance ID, and its addition, change, or removal is the border between messages

This problem is already solved more robustly by using separate streams per-message.

This makes divisions between messages more robust and message transmission faster, as it lets messages on the same channel be sent one after another without having to wait too long for acknowledgements of resets to come back.

It's unclear what a "channel" is. Sending unordered messages as separate streams does not require waiting for STREAM_RST acknowledgement, whether or not you should choose to send some. Sending ordered messages within a single stream does not involve sending STREAM_RSTs at all. QUIC goes to some effort to ensure you never have to wait for a round-trip before proceeding unless the receiver wants you to.

Because packet delay is a thing, short delays should be put between transmissions of messages on the same channel.

Inter-packet timing is not preserved across the network.

Messages which are not small enough to fit in a single frame but are less than about 10 frames in length are encoded through STREAM_IMPLICIT_RST frames, if available.

What does this do that sending the message as a stream doesn't?

Ultimately, it's not clear what problem this extension trying to solve. A QUIC extension should be motivated by enabling something that is otherwise difficult or impossible. If your problem can be solved effectively without inventing a new extension, that should be preferred, because you are hence able to maximize your benefit from the engineering that has already been done, leave open the door for interoperability, and reduce opportunity for error.

[fhaynes]

Note that we are not ignoring this. We want to finish 0.1 before we dive into this. Once that is done, QUIC and proposed extensions will receive a lot of attention. The amount of work ya'll have put in will be invaluable, thank you for that.

[TimonPost]

@Ralith can you elaborate more on the option of "It should be noted that QUIC can easily be applied out-of-the-box for any combination of reliable/unreliable ordered/unordered messaging" I can't directly find more information about this part of QUIC."

Can I choose if QUIC sends packets: unreliable or reliable, ordered or unordered, sequenced or just reliable?

[Ralith]

QUIC does not send packets reliably or ordered; it retransmits information so long as it is relevant, always composing new packets to do so, and the application can declare a certain outgoing stream of data to no longer be relevant at any time by "resetting" it. Hence, you can take the following approaches at the application layer to implement every type of messaging:

• Reliable + Ordered: Send a single stream with application-level framing, as you would in TCP
• Reliable + Unordered: Create a new stream for every message. Streams are free and instantaneous to construct so long as you do not exceed the peer's flow control limits, and have compact framing, so even large numbers of streams sending small messages will work well.
• Unreliable + Ordered: Create a new stream for every message, tagging them at the application level with a sequence number and discarding messages which do not increase the highest sequence number at the receiver. If large messages are expected, a "stop sending" command can be issued at the transport level by the receiver to force the sender to abandon the stream. If very small messages are needed, the partial reliability extension could be used to preclude the need for an application-layer sequence number.
• Unreliable + Unordered: Create a new stream for every message, and reset them when they are no longer relevant. For example, a game server could reset outgoing streams containing world state updates for a old timestep after it has transmitted data for a newer one, either immediately or after it has been acknowledged by the receiver. Alternatively, you could reset outgoing streams on a timer. A QUIC implementation could also support implicitly resetting streams after transmitting their data a single time without requiring any changes to the protocol, if necessary.

For every form of unreliable messaging, you also have the option to abandon streams in favor of the QUIC datagram extension, which lets you transmit smaller-than-MTU data without fragmentation or a need to explicitly reset, while still retaining the benefits of operating within a QUIC connection (i.e. the handshake, flow control, congestion control, multiplexing, and security measures). I expect this to only very rarely be necessary.

Note that "application layer" above (and in QUIC documentation in general) refers to the layer above the QUIC implementation, whether that's a literal application or another library providing domain-specific abstractions.

[TimonPost]

Thanks for taking the time to answer, so, if I'm understanding you correctly, laminar - at the application level - 'could' wrap QUINN and provide some of the above-noticed delivery types?

I've got twho other questions:

1: Are you familiar with ordering stream and would this be possible with QUINN/QUIC?
2: I saw TLS 1.3 support in QUIN are you doing your own TLS-kind of implementation? Since there are crates for TLS 1.3 however QUINN is not using it. And what is the status of the TLS 1.3 implementation?

[Ralith]

laminar - at the application level - 'could' wrap QUINN and provide some of the above-noticed delivery types?

Yes, these strategies are available to anyone calling into a QUIC implementation.

Are you familiar with ordering stream and would this be possible with QUINN/QUIC?

Yes, you can use the above strategies to implement whatever ordering relationships you like between your messages. For example, to have two independently ordered reliable message streams, you would use two QUIC streams with application framing--that's what they're for.

I saw TLS 1.3 support in QUIN are you doing your own TLS-kind of implementation? Since there are crates for TLS 1.3 however QUINN is not using it. And what is the status of the TLS 1.3 implementation?

Current IETF QUIC mandates that all conforming implementations use a specific cryptographic protocol, which is an adaptation of TLS 1.3. Quinn uses rustls, a Rust-native TLS 1.3 implementation, for the bulk of this logic. The implementation is complete. Some patches to rustls were required, but are expected to be merged before the upcoming major release; see rustls/rustls#187 for details.

[TimonPost]

Really thanks for helping out, last but probably not the least question, do you really think QUIC is a good fit for a high-performance game network protocol since it is originally developed for speeding up the web communication? I did a lot of research into QUIC last days however I'm probably not as updated as you are on how the internals work. And are there any weaknesses of QUINN that might be helpful to know in front
of even thinking about using QUINN as a protocol implementation?

[Ralith]

do you really think QUIC is a good fit for a high-performance game network protocol since it is originally developed for speeding up the web communication

QUIC has always been designed as a general-purpose protocol. While it will undoubtedly be a great boon for the web, the problems it solves are general problems, and it provides general solutions. In fact, my motivation for developing Quinn is specifically to have a secure, robust, and easy to use transport protocol for my own real-time multiplayer game projects.

And are there any weaknesses of QUINN that might be helpful to know in front of even thinking about using QUINN as a protocol implementation?

• Implementation work on Quinn has been focused almost entirely on correctness so far, so CPU and memory load have room for improvement. This is unlikely to be noticeable for typical game applications where total data rate is low, as opposed to bulk data transfer. Efficient resource use is a goal of the project, regardless.
• Quinn's high-level public interface is asynchronous and futures-based, which means it will undergo significant changes upon the stabilization of std futures and async/await in order to capture their ergonomic benefits.
• Quinn is young; there will be bugs, and the interface is unstable. While it should save a lot of time compared to developing similar capabilities from scratch, there will be rough spots. That said, development is quite active, and bugs affecting real-world use will be a priority.

[LucioFranco]

@Ralith thank you so much for this great information. I took a look at quinn this morning and it's looking like its coming along really well (excited for quinn-h3).

I think our biggest blocker for using quinn right now the fact that it is based on tokio and not directly on mio. I think this will impede how it could possibly get integrated into laminar in its current state. That said, I had this thought that in the future, once the whole quic, futures and tokio stuff all gets settled laminar could just be a proto on top of quinn. I think that might be a great use of this.

I am really really excited to see where quinn goes and I hope that we can have a strong relationship and work together on a really good transport protocol for real time games :). I can totally see even amethyst moving off of laminar in favor of some quinn based multiplayer protocol.

[Ralith]

In principle, it is possible to use quinn-proto with mio directly; the core protocol logic is totally agnostic to your event loop. You drive it by passing in received packets, querying the state machine for packets to send and timers to start/stop, and passing application-facing events outward. However, quinn-proto's API is even more unstable than quinn's, and async/await will be much more ergonomic, so waiting for the ecosystem to settle a bit there makes sense.

[LucioFranco]

@Ralith thats great to hear I guess I had flipped the level of abstraction that proto does. We are in the process of migrating laminar to mio so I think that down the line that might be a really good fit once it becomes a bit more stable.

[TimonPost]

QUIC allows you to have your own congestion control algorithms, does QUINN also provide that and or does QUINN have some already implemented?

[Ralith]

Quinn currently uses the congestion control algorithm specified in the IETF recovery draft, and fully supports explicit congestion notification. Pluggable congestion control would be interesting to support should a need arise.

[TimonPost]

We did decide a while ago that for now, QUIC is not really a solution for us.

We wrote a document on this:

https://docs.google.com/document/d/1ypUNgELQ7jpxEEyjsWb7QBcXCUIg_hWPBGOQSzT0as4/edit?usp=sharing

Conclusion:

There are a few things needed to be done before we can switch to QUIC

1. QUINN and QUINN-proto should be at least somewhat stable.
2. Amethyst-engine should be comfortable with Tokio and Future’s
3. We should have performance and benchmark tests of QUINN compared to laminar.
4. We should first move laminar to MIO before migrating to QUINN.
5. Figure Out if all delivery options are realistic to implement.
6. QUICK is built for the web.

The last one is the most speculative.

"I am a bit skeptical of some of the reliability options of QUIC, for unreliable you could Create a new stream for every message and reset them when they are no longer relevant. However, this does not seems optimal to me."

[Ralith]

QUICK is built for the web.

This is wrong. The IETF has taken care to ensure that QUIC is a general-purpose protocol, and they succeeded. HTTP/3 is built for the web; QUIC is not HTTP/3.

" ... However, this does not seems optimal to me"

It's difficult to respond to such a vague criticism. Opening and resetting streams are both computationally trivial operations that have zero latency. This very issue also contains discussion of the simple QUIC datagram extension, which allows you to work with datagrams that have reliability similar to raw UDP packets, if for some reason this is deemed necessary. In practice, it should not be necessary except as a minor optimization for cases where both loss and reordering are acceptable.

[TimonPost]

There was some circle in arguments but we, as a team, decided to not go for QUIC a long time ago. Thanks for putting your arguments here, when it is needed, we can always revisit it. But for now, it will be closed.