Lossy protocols utilizing datagrams

[kixelated]

Hey folks,

We've investigated using QuicTransport datagrams to implement our own protocol for lossy delivery. For simplicity, let's assume that our protocol uses acknowledgements to selectively retransmit packets. Other protocols like RTP will use negative acknowledgements but it's a similar concept.

The first issue is that QUIC datagrams themselves are ack-eliciting. The application needs its own acknowledgement mechanism as these QUIC acknowledgements are not exposed to the application. This doubles the number of packets in the worst case, although any decent QUIC implementation will coalesce multiple frames.

----> DATAGRAM (DATA)
<---- ACK
<---- DATAGRAM (ACK) 
----> ACK

The second issue is that QUIC datagrams are congestion controlled and will be dropped or queued. The application needs its own congestion control algorithm as these limits are not exposed to the application. This limits the maximum throughput to the lower bound of both algorithms, which may interact in non-obvious ways.

[kixelated]

I filed #105 as a potential solution for the latter, although I now realize that I should have explained the problem and not a solution.

[LPardue]

The DATAGRAM I-D says:

If a sender detects that a packet containing a specific DATAGRAM
frame might have been lost, the implementation MAY notify the
application that it believes the datagram was lost. Similarly, if a
packet containing a DATAGRAM frame is acknowledged, the
implementation MAY notify the application that the datagram was
successfully transmitted and received.

If the WebTransport API fulfilled this requirement, would your problems be addressed? It seems a shame to require apps to reimplement something that the transport is doing.

[murillo128]

I think that @kixelated described perfectly the main issues for implementing a loosy protocol over quic/datagrams.

The datagram ACK/NACK could be solved by exposing the events back to the app as the DATAGRAM i-D recommends.

I am much more worried about the interaction between the bandwidth estimation, congestion control and the queueing/frame dropping. Maybe it would be good to split the issue in two for better tracking.

[vasilvv]

The first issue is that QUIC datagrams themselves are ack-eliciting. The application needs its own acknowledgement mechanism as these QUIC acknowledgements are not exposed to the application.

This was an intentional design decision, see #23. The problem is that even when the QUIC stack has issued an acknowledgement to the peer, it does not mean that the datagram in question will necessarily reach the application, since it still can be dropped if the datagram receive queue is full. This is the similar reason as to why application protocols like QPACK have their own ACKs: processing on lower layer does not imply processing on a higher layer.

The second issue is that QUIC datagrams are congestion controlled and will be dropped or queued. The application needs its own congestion control algorithm as these limits are not exposed to the application. This limits the maximum throughput to the lower bound of both algorithms, which may interact in non-obvious ways.

This is issue #21. We had some ideas on how to approach this, but did not want to start working on this without any Web developers who are specifically interested in this.

[murillo128]

IMHO having the ACK status of each datagram is interesting even it is not guaranteed that the datagram is processed by the js app layer.

Taking webrtc as a model, current NACK mechanism does not ensure that the frame is decoded by the receiver side, it may happen that the cpu is not powerful enough and the frame has to be dropped before decoding even if has been correctly received.

So I would foresee having different ACK/backpressure info depending on the application, but I would prefer avoiding to have to duplicate network info that is already available and could be exposed to the js layer.

Last, I don't buy the argument of being a footgun, as the whole webtransport datagram is already a footgun if not handled carefully ;)

[kixelated]

It's nice to see that there's been other threads!

@vasilvv You make a good point that a QUIC ACK signals that a packet has been received, but not necessarily processed. This would not be sufficient for some protocols (ex. QUIC), although it may still be useful for other protocols (ex. RTP, TCP). This caveat would need to be very explicit if these acknowledgements were exposed.

As for datagrams getting dropped prior to reaching the application layer... I'm not sure if that's the right behavior, at least I don't think it matches UDP. As far as I know, UDP packets will be buffered indefinitely while there is buffer space otherwise new packets are dropped. Peter makes it seem like older datagrams should be dropped instead, but I think that's up for discussion. The real issue is the lack of flow control for datagrams which allows this buffer to overflow.

[vasilvv]

As for datagrams getting dropped prior to reaching the application layer... I'm not sure if that's the right behavior, at least I don't think it matches UDP. As far as I know, UDP packets will be buffered indefinitely while there is buffer space otherwise new packets are dropped. Peter makes it seem like older datagrams should be dropped instead, but I think that's up for discussion. The real issue is the lack of flow control for datagrams which allows this buffer to overflow.

There's some question as to whether we want to drop from head of the queue or from its tail, and the exact criteria on when to drop. Regardless, even if it behaves identical to the kernel UDP queue, the QUIC ACK signal is not helpful, since all of the datagrams dropped due to the queue being full still have to be acknowledged at the QUIC layer (because (1) packets can contain a mix of DATAGRAMs and reliable frames, and (2) peer will interpret missing ACKs as a loss signal, and throttle the connection).

[kixelated]

Hmm yeah that makes sense. So you would need datagram flow control to expose ACKs, and even then it wouldn't be particularly useful.

[aboba]

We did look at exposing datagram acknowledgements, but as @vasilvv noted, application layer acknowledgement would be needed anyway. BTW, the same conclusion can be reached with respect to WebRTC data channel and WebSockets "lack of backpressure" - that you need an application-layer acknowledgement so that the sender can throttle back when the receiver is unable to empty the JS event queue fast enough, though the receive window is (deceptively) open.

[jan-ivar]

Meeting:

• We did look and the answer was no. Even if you add the info it doesn't tell you the app got it, so you need the application ACK anyway. Closing.

[aboba]

From RTP over QUIC Section 4.1:

" If the used QUIC implementation is not directly incorporated into the
RTP over QUIC mapping implementation, it has to fulfill the following
interface requirements. The QUIC implementation MUST support QUICs
unreliable datagram extension and it MUST provide a way to signal
acknowledgements or losses of QUIC datagrams to the application.
Since datagram frames cannot be fragmented, the QUIC implementation
MUST provide a way to query the maximum datagram size, so that an
application can create RTP packets that always fit into a QUIC
datagram frame."