RTCSctpCapabilities maxMessageSize issues

[lgrahl]

I see two problems with the maxMessageSize attribute of RTCSctpCapabilities:

1. An unsigned short provides insufficient space. I'm not entirely sure what the maximum message size of SCTP effectively is, maybe @tuexen can add a statement to that. However, streaming APIs should be able to handle messages of arbitrary size (although the W3C API is not a streaming API, other implementations may provide one).
2. We should clarify that 0 indicates that the other peer can handle messages of any size. This is particularly important once SCTP ndata will be deployed.

For reference, here is the SDP section relevant for WebRTC.

[aboba]

In WebRTC 1.0, the maxMessageSize attribute is an unsigned long, so there is a potential compatibility issue:
https://rawgit.com/w3c/webrtc-pc/master/webrtc.html#rtcsctptransport-interface

[lgrahl]

unsigned long means uint64_t right? That should be sufficient. :)

WebRTC 1.0 also does not clarify the 0 case. Maybe we should ping them about that as well.

[tuexen]

On 17 Dec 2016, at 17:11, Lennart Grahl ***@***.***> wrote: I see two problems with the maxMessageSize attribute of RTCSctpCapabilities: • An unsigned short provides insufficient space. I'm not entirely sure what the maximum message size of SCTP effectively is, maybe @tuexen can add a statement to that. However, streaming APIs should be able to handle messages of arbitrary size (although the W3C API is not a streaming API, other implementations may provide one).

SCTP has no message size limit. The protocol can handle messages of arbitrary size. However, since buffers are finite, it uses partial delivery at the receiver side and an explicit EOR mode on the sender side, if you follow the SCTP socket API specified in https://tools.ietf.org/html/rfc6458. The JS APIs doesn't support this, so they need to buffer the complete messages. Therefore the limit is actually a limit enforced by the code running on top of the SCTP implementation. Best regards Michael

…

• We should clarify that 0 indicates that the other peer can handle messages of any size. This is particularly important once SCTP ndata will be deployed. For reference, here is the SDP section relevant for WebRTC. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[lgrahl]

Thanks Michael. Taking into account that implementations can set maxMessageSize to 0 to indicate support for messages of arbitrary size, having a 64 bit unsigned integer should be absolutely fine. Implementations that choose to limit message sizes will usually choose a much lower amount than 2^64 anyway (but it's good to have the full range from 1 to 2^64).

[aboba]

Proposed resolution: change to unsigned long

[lgrahl]

And maybe adding something like this to the description:

A value of 0 indicates that messages of arbitrary size can be handled.

[robin-raymond]

@lgrahl if EOR is not supported, maxMessageSize is limited to the max SCTP packet size and arbitrary sizes cannot be supported. Even if this limitation was fixed, message sizes are still not unlimited due to other constraints (e.g. memory). Even TCP has max message sizes, but it gets around this by sends of some of the buffer rather than all of the buffer, but we can't do that here because the buffers are all or nothing. Thus I prefer to have a max size without a "0" to a) not require an additional "if (0 == maxMessageSize)" checks and b) there's always realistically going to be a maxMessageSize. I'm okay with a large possible value though up to 2^64.

[lgrahl]

I do understand that this is a limitation of ORTC's design as we do not have a streaming API. Yes, I agree that 2^64 is more than enough, don't get me wrong. But 0 is definitely not a valid maxMessageSize as SCTP cannot handle empty messages, so it needs to be handled differently anyway, right? So, I don't understand your point that additional checks would be required.

0 would be a crystal clear indication for fragmentation/reassembly projects such as https://github.com/saltyrtc/chunked-dc-js that no chunking is needed at all. While 2^64 can be interpreted as unlimited, everything below that is a question mark: Is it large enough for my use case or do I need to turn on fragmentation/reassembly?

Furthermore, I probably should have mentioned earlier that draft-ietf-mmusic-sctp-sdp does handle the 0 case.

Edit: Sorry for all the editing.

[aboba]

@lgrahl @pthatcher @robin-raymond In looking at the usage of maxMessageSize within the start method and example code, some questions arise. As defined in WebRTC 1.0, maxMessageSize represents a limitation on the use of the send() method, for the application developer to take into account. However, in Example 22, maxMessageSize is exchanged in signaling and the start method is called with remoteCaps as an argument. Passing the remote maxMessageSize to the start method would seem to imply that the local implementation will do something with this information, but it isn't clear to me what this should be.

[aboba]

@lgrahl @pthatcher While draft-ietf-mmusic-sctp-sdp does handle 0 as unlimited, is this something that implementations support? Given that Binary Partial and String Partial are both deprecated within the SCTP parameters registry (see http://www.iana.org/assignments/sctp-parameters/sctp-parameters.xhtml#sctp-parameters-25), I don't see how this is expected to work.

[lgrahl]

@aboba I'm not sure if I understand your question correctly, but basically this value needs to be stored and checked in send calls. This also means that RTCDataChannel.send is not available before the SCTP transport is in the connecting state. But this isn't a problem as the data channels wouldn't be open at that point.

Regarding your second question: Yes, implementations can support messages of arbitrary size by using SCTP's explicit EOR mode when sending and checking for the EOR flag when receiving. In fact, our implementation already supports messages of arbitrary size. Sadly, this hasn't been implemented in browsers as both Firefox and Chromium ignore the EOR flag completely which is a bug and an open issue @tuexen and I have discussed several times. Basically, @tuexen didn't want to open an issue before we have a patch. This bug is also the reason why we cannot send more than 64 KiB reliably (16 KiB in FF - Chromium interop, I've written about this in detail here) until it has been resolved.

[lgrahl]

@robin-raymond I overlooked your point about no EOR support. It's vital that we know whether we talk about EOR when receiving or EOR when sending because one is required and one is not. Every SCTP-based data channel implementation must look at the EOR flag when receiving messages or it will violate the message-oriented principle of SCTP and data channels. This is exactly what pretty much every implementation out there currently does (violate). You've mentioned that TCP supports arbitrary sized messages by sending only parts of the buffer. It's the same for SCTP which will also send only parts of the buffer and not all or nothing. That's why the EOR flag exists and why looking at it is so essential even if an implementation doesn't use the explicit EOR mode for sending.

Back to the main topic: Granted, by allowing maxMessageSize to be 0, there's one further if (remote.maxMessageSize == 0) conditional required in the send method and potentially in the receive/reassembly function. Honestly, I don't think that's a big deal. We'd trade that for the conditional required when validating the RTCSctpCapabilities.maxMessageSize because if 0 is not unlimited, then 0 is not acceptable as SCTP is not capable of sending messages of size 0. Furthermore, let's not forget that ORTC is currently incompatible to draft-ietf-mmusic-sctp-sdp. A WebRTC-ORTC shim will have to compensate if we leave it as is.

[lgrahl]

Ping! Any new thoughts/comments?

[lgrahl]

By the way, we have a working implementation that can handle messages of arbitrary size and supports maxMessageSize to be 0. The size checks were as simple as that: https://github.com/rawrtc/rawrtc/blob/7fd3aa3d2707f3f8d844ef3a4ec201d9b8f43d12/src/librawrtc/sctp_transport.c#L2449

[robin-raymond]

I think the biggest issue is going to be the lack of support for EOR and backwards compatibility until it becomes prolific. We may need an EOR usage setting for compatibility. Also does there need to be any guidelines to how to handle larger message that are received but still incomplete due to missing segments of the larger message in a lossy sctp mode? I'm not sure if this is covered via RFC(s) already or not or if it's possible to have more than one larger messages being reassembled at one time with missing segments of the larger message still outstanding.

[lgrahl]

I think the biggest issue is going to be the lack of support for EOR and backwards compatibility until it becomes prolific. We may need an EOR usage setting for compatibility.

I don't see any backwards compatibility issues. Handling EOR and reassembling messages when receiving isn't optional, it never was. Let's dig into SCTP's RFCs for an explanation:

RFC 4960, section 6.9. states

An endpoint MAY support fragmentation when sending DATA chunks, but
it MUST support reassembly when receiving DATA chunks.

The SCTP API (RFC 6458) clarifies that this has to be done by the application to some extent (I don't know why this section isn't mentioned for sctp_recvv but the mechanism stays the same)

If the SCTP stack is running low on buffers, it may partially deliver
a message. In this case, MSG_EOR will not be set, and more calls to
recvmsg() will be necessary to completely consume the message.

In case of usrsctp, if you use the callback API, a call to recvmsg() is virtually the same as a call to the callback function. So it's the job of the application to look at the EOR flag. Frankly speaking, it's a bug in Mozilla's and Google's data channel code that they don't handle the flag.
(@tuexen: We should really get a move on and report this to Mozilla and Google. I don't think we need a patch for that. We can link them the corresponding line in RAWRTC's code as reference to show how this can be implemented.)

Explicit EOR when sending is an API mode that can be activated in the SCTP API (see RFC 6458, section 8.1.26 and has no implications on compatibility.

---

Also does there need to be any guidelines to how to handle larger message that are received but still incomplete due to missing segments of the larger message in a lossy sctp mode? I'm not sure if this is covered via RFC(s) already or not or if it's possible to have more than one larger messages being reassembled at one time with missing segments of the larger message still outstanding.

That question is partly answered by section 3.1.4 in the SCTP API (RFC 6458)

Only
one message at a time can be partially delivered in any stream. The
socket option SCTP_FRAGMENT_INTERLEAVE controls various aspects of
what interlacing of messages occurs for both the one-to-one and the
one-to-many style sockets.

More precisely, RFC 6458, section 8.1.20 allows the use of three different levels which control the application's desired interleave behaviour. Note that this does not change SCTP's behaviour - there's nothing to negotiate here. It simply controls how the API behaves towards the application. Broadly speaking level 0 prevents interleaving of messages when receiving, level 1 allows interleaving from different associations when receiving and level 2 allows interleaving of messages from different streams but message chunks will not be interleaved, so you will always receive pending chunks for one message on a stream until it is complete. AFAIK level 2 has no effect until the SCTP ndata extension has been deployed and negotiated (but it doesn't hurt if your application can already handle interleaved messages on different streams).

IIRC when message chunks of a message have already been handed out but there will be no retransmissions for the remaining chunks, the SCTP_PARTIAL_DELIVERY_EVENT will be raised as described in RFC 6458, section 6.1.7.

---

Implementations that can send messages of arbitrary size still have to obey the maximum message size when sending, so I don't see an incompatibility there either. The default maximum message size in WebRTC's SDP is 64 KiB which is exactly what current implementations (with the bug mentioned above) can handle.

(@tuexen Please correct me if I'm wrong with any of my statements above.)

[robin-raymond]

@lgrahl I'm well aware EOR is not optional, but sadly it was not implemented correctly everywhere so to assume it exists at the moment is problematic for compatibility in the real world so while we should fix the spec, this is an ongoing concern; i.e. I do agree that we should have a max message size as an unsigned long (although I don't necessarily agree with the meaning of "0" as unlimited, I'd rather the value be "unset").

As for the fragment interleave, that's exactly what I was concerned about - what surface control do we need to expose to the application to control the expected behaviour of the engine. Do you have specific proposal for this?

[lgrahl]

Let's put the discussion about 0 aside for a moment. What exactly is the compatibility issue we face at the moment? In the ORTC API the maxMessageSize capability is not an optional field - you have to set it. What's the problem if current implementations simply use whatever is applicable for them? 64 KiB for pretty much every implementation out there that doesn't care about EOR when receiving and 16 KiB for Firefox because that's the chunk size they use with the deprecated fragmentation/reassembly mode based on PPIDs. That's what I'm defaulting to in our WebRTC-ORTC shim and it works great in my RAWRTC browser interoperation testing.

As for the fragment interleave, that's exactly what I was concerned about - what surface control do we need to expose to the application to control the expected behaviour of the engine. Do you have specific proposal for this?

I don't think I understand what you mean. Can you rephrase? What's application and what's engine for you? Also, what kind of proposal do you mean - I guess for the ORTC spec? I'm a bit confused, sorry. :)

[robin-raymond]

The EOR issue is unrelated to the maxMessageSize. My point was that allowing a larger max message size indirectly brought up the issue that EOR is a problem for compatibility in the real world at the moment because of lack of compliance. Any API that doesn't offer an override to disable EOR may cause interaction problems with non-compliant remote parties even though "must" is specified. You may be right, the remote party indicating a low maxMessageSize might be sufficient to never cause the EOR incompatible issue to become a problem when connecting an EOR compliant implementation with the an non-compliant remote party. Certainly the spec should assume all parties are compliant but ... the reality is they aren't.

Application = anything high layer calling an implementation of the ORTC spec. Engine = any implementation of the ORTC spec.

[lgrahl]

Ah, okay. I don't see a need for an EOR on/off switch as it's working perfectly fine if maxMessageSize = 65536 is chosen for non-compliant engines. I'll explain by going through both cases: Sending a mesage from an EOR-compliant engine to a non-compliant engine and vice versa.

Sending from EOR-compliant to non-EOR-compliant

EOR-compliant engines can only send 64 KiB to non-EOR-compliant engines. The reason why 64 KiB works is because usrsctp's default partial delivery point is 64 KiB and no engine I've seen so far adjusts this variable. That's why usrsctp delivers 64 KiB chunks to the engine. During the development of RAWRTC (which always uses EOR), I've done extensive tests with Chromium and Firefox and I have seen no issues when sending 64 KiB towards Firefox and Chromium.

Receiving from non-EOR-compliant to EOR-compliant

Let's summarise existing compatibility issues for non-EOR-compliant engines that talk to each other before we talk about EOR-compliant engines: All existing WebRTC engines I know ignore or don't even provide the maximum message size attribute in the offer/answer SDP the last time I checked. Chromium can send up to 256 KiB, everything above raises an error and closes the data channel. But when receiving on either Chromium or Firefox, this 256 KiB message appears as four different 64 KiB messages because these engines don't look at the EOR flag. Furthermore, Firefox can send messages of arbitrary size and uses the deprecated fragmentation/reassembly mode based on PPIDs. And guess what, Chromium doesn't support that mode and delivers the chunks to the application as if they were different messages. That's the 16 KiB Firefox-Chromium interop limitation in a nutshell. So, these two browsers are already highly incompatible to one another with lots of corner cases if you send more than 16 KiB sized messages. Chromium is even incompatible to itself. The whole message-oriented principle shredded to pieces. (Sorry for being dramatic :))

EOR-compliant engines provide the maximum compatibility because they a) reassemble the 256 KiB messages that Chromium sends and b) can add the deprecated fragmentation/reassembly mode as it is very simple to add if the engine is already EOR-compliant. This provides compatibility with Firefox.

---

So, to sum it up: If EOR-compliant engines provide much better compatibility to existing engines than non-compliant EOR engines. Why have a switch in ORTC to turn if off?

[lgrahl]

I'm working with the guys at Mozilla to bring EOR into Firefox's WebRTC data channel implementation. Maybe we can convince you with a real world example that EOR doesn't pose a threat to compatibility. Will keep you posted.

[lgrahl]

So, this is the plan we have worked out for Firefox regarding EOR (keep in mind, I'm talking about WebRTC here but I will explain the effect it will have for ORTC interop):

1. Add EOR support for both sending and receiving.
2. Set a=max-message-size in the SDP (we haven't discussed to what value yet but everything up to 2^32 - 1 is possible - the code has some limitations which is why we cannot use 0 at the moment).
3. Check the size of the message before sending. If len(message) > maxMessageSize, raise an error in the send method. (This is currently not standardised, but I have opened Sending data channel messages > maxMessageSize webrtc-pc#1205. Once it has been resolved, we should update the ORTC spec as well. It has been standardised implicitly by Throw error if data channel's buffer is filled, rather than closing. webrtc-pc#1209 - raise TypeError)
4. And one further Firefox-specific compatibility issue: a) Reassemble messages that have been fragmented using the deprecated partial PPIDs, and b) Use PPID-based fragmentation when sending in case an older Firefox browser has been detected. Important: This is purely Firefox-specific as no other browser to my knowledge supported this. It will be removed eventually and I'm only mentioning this for the sake of completeness.

Compatibility: EOR & non-EOR

• Message from non-EOR to EOR: Messages will be reassembled properly, even from current implementations who unintentionally fragmented messages > 64 KiB. (Adding PPID-based reassembly is extremely easy to implement if EOR-based reassembly is already done. This will further improve compatibility with older Firefox browsers.)
• Message from EOR to non-EOR: If the remote peer does not set a=max-message-size, it defaults to 65536 (specified in SDP). All browsers currently do not set a=max-message-size and are capable to receive messages up to 65536 bytes. The check in 3. ensures we cannot send messages that would break message integrity.

Compatibility: WebRTC-ORTC Shims & API

• WebRTC-ORTC shims simply have to translate a=max-message-size properly. If it's not set, maxMessageSize = 65536 in the remote SCTP capabilities.
• Existing implementations that have no EOR support need to set their local SCTP capabilities maxMessageSize = 65536.
• Once we have added the check explained in 3. to both the WebRTC and the ORTC API, we have resolved all compatibility issues.

Are there open questions regarding EOR?