MTU handling

[lancerchao]

After chatting with Fridolin and reading RFC 6347, I notice a discrepancy between the specs and MTU handling in KTLS, which I would like to discuss.

Right now in KTLS, if the MTU is 1500, and the user tries to send a message of length 1501 over UDP, only 1500 bytes is transmitted in order to avoid IP fragmentation. In other words, KTLS is actively cutting down the size of records to avoid fragmentation.

However, RFC 6347 says this.

In general, DTLS's philosophy is to leave PMTU discovery to the application.

It's saying that the application using DTLS is responsible for avoiding IP fragmentation, not the DTLS implementation itself.
Additionally, it goes on to say:

However, DTLS cannot completely ignore PMTU for three
reasons:

• The DTLS record framing expands the datagram size, thus lowering
  the effective PMTU from the application's perspective.
• In some implementations, the application may not directly talk to
  the network, in which case the DTLS stack may absorb ICMP
  [RFC1191] "Datagram Too Big" indications or ICMPv6 [RFC4443]
  "Packet Too Big" indications.
• The DTLS handshake messages can exceed the PMTU.

As a solution to the above problems, it says DTLS should provide the following:

If PMTU estimates are available from the underlying transport
protocol, they should be made available to upper layer protocols. In
particular:

• For DTLS over UDP, the upper layer protocol SHOULD be allowed to
  obtain the PMTU estimate maintained in the IP layer.

My recommended changes:

• Remove setsockopt(MTU)
• Revise getsockopt(MTU) to query the MTU from the IP layer, and return MTU - KTLS_DTLS_OVERHEAD to the user
• Do not truncate records that above MTU size length. Instead, allow fragmentation to occur.

[nmav]

Revise getsockopt(MTU) to query the MTU from the IP layer, and return MTU - KTLS_DTLS_OVERHEAD to the user

How would you obtain the MTU from the IP layer?

Do not truncate records that above MTU size length. Instead, allow fragmentation to occur.

I think that is a reasonable approach.

[lancerchao]

How would you obtain the MTU from the IP layer?

I'm not too sure. It looks like TCP has some kind of "PMTU probe." but I think that may be too complicated for our purposes. Instead, I think we need only implement this:

The DTLS record layer SHOULD allow the upper layer protocol to
discover the amount of record expansion expected by the DTLS
processing. Note that this number is only an estimate because of
block padding and the potential use of DTLS compression.

We can change setsockopt(MTU) to take a MTU and return MTU - KTLS_DTLS_OVERHEAD from getsockopt(MTU), and let userspace decide how big the records should be.

[fridex]

Right now in KTLS, if the MTU is 1500, and the user tries to send a message of length 1501 over UDP, only 1500 bytes is transmitted in order to avoid IP fragmentation. In other words, KTLS is actively cutting down the size of records to avoid fragmentation.

The current implementation is performing such thing for sendfile(2) (kernel_sendpage() respectively) for both TLS/TCP and DTLS/UDP: https://github.com/ktls/af_ktls/blob/master/af_ktls.c#L318 It was implemented since we were testing it with splice(2).

Note however that UDP (DTLS) is quiet tricky in this case. If we split a message "AABB" into "AA" + "BB" because of MTU transparently to user space, a receiver can read "BBAA" since DTLS is not dealing with out of order delivery.

Personally I would stick with returning an error code (such as E2BIG) and let user space deal with possible fragmentation (which could be in many cases application specific) - the user space should explicitly know about the risk here. IIRC, GnuTLS is doing something similar.

In general, DTLS's philosophy is to leave MTU discovery to the application.

That's also an argument why we should leave such setsockopt(2) call.

... and let userspace decide how big the records should be.

Yes, that was the purpose of letting userspace to set MTU (according to possible MTU discovery).

[nmav]

Note however that UDP (DTLS) is quiet tricky in this case. If we split a message "AABB" into "AA" + "BB" because of MTU transparently to user space, a receiver can read "BBAA" since DTLS is not dealing with out of order delivery.

That's correct, but note that Lance was proposing sending the larger packet as a singleton and let IP framentation work in the middle routers. The end host will defragment such packets and thus the whole packet will be received in the correct order/size by the peer.

However letting the lower layer know is important as you mention. Currently in UDP this is done by applications setting the MTU discovery setsockopt() in the fd, informing the kernel to return EMSGSIZE if something is larger than the known MTU. I guess we could rely on the userspace app to set that option to the original fd, and propagate any error? For sure we need quite some tests around that.

[djwatson]

I'm in favor of removing setsockopt MTU, and just dealing with it using MSG_MORE flag

[fridex]

Setting MTU was useful mostly for sendfile() where you get up to page size bytes in a one call. Without setting MTU it would be not possible to use DLTS - you would send up to 4096B+DTLS overhead in a one single UDP datagram, which is not doable on most networks.

[lancerchao]

Without setting MTU it would be not possible to use DLTS - you would send up to 4096B+DTLS overhead in a one single UDP datagram, which is not doable on most networks.

I don't understand why not? IP fragmentation exists for the purpose of sending large datagrams. UDP can send up to 65k messages using fragmentation, and like Nikos said, the end host can defragment the packets and thus the application layer receives packets as a whole. See the frag_list field in struct sk_buff.

[fridex]

Ah, ok. We can then use IP fragmentation and get rid of MTU setsockopt as Dave pointed out.