Messages sent without MSG_EOR set end up with MSG_EOR set server side

[dirkz]

Use case

Sending arbitrarily big messages to the server in parts. Every part sent is smaller than the sb_free advertised to send_cb, and has flags = 0. The last message has MSG_EOR set.

Expected

The server (in receive_cb) receives all message parts in order and with flags = 0, except the last message, which has MSG_EOR set. The SSN only gets incremented after messages with MSG_EOR set.

Observed

Individual message parts arrive with MSG_EOR set, and the SSN is incremented for every message part the server receives.

Platform

I can reproduce this on macOS 11.5.2 (big sur), xcode 12.5.

Notes

See sample code forked from master here: https://github.com/dirkz/usrsctp/blob/arbitrary_size/programs/client.c.

Sample output:

Client:

usrsctp (arbitrary_size)$ ./Build/programs/client ::1 7 0 324 423
*** Sent 4096 bytes (262144 available), completed 0
handle_notification : SCTP_ASSOC_CHANGE
Association change SCTP_COMM_UP, streams (in/out) = (10/10), supports PR AUTH ASCONF MULTIBUF RE-CONFIG.
Local addresses: 192.168.1.35, ::1, 127.0.0.1.
handle_notification : SCTP_PEER_ADDR_CHANGE
Peer address 192.168.1.35 is now SCTP_ADDR_CONFIRMED (error = 0x00000000).
Peer addresses: ::1, 192.168.1.35, 127.0.0.1.
handle_notification : SCTP_PEER_ADDR_CHANGE
Peer address 127.0.0.1 is now SCTP_ADDR_CONFIRMED (error = 0x00000000).
*** Sent 4096 bytes (259252 available), completed 0
Could not send bytes: Connection reset by peer
Could not send bytes: Connection reset by peer
Could not send bytes: Connection reset by peer
Could not send bytes: Connection reset by peer
Could not send bytes: Connection reset by peer
handle_notification : SCTP_ASSOC_CHANGE
Association change SCTP_SHUTDOWN_COMP, streams (in/out) = (10/10).
Number of packets (sent/received): (22/21).

Server:

Unchanged from the current master. Same behavior with the discard server.

usrsctp (arbitrary_size)$ ./Build/programs/echo_server 423 324
Notification of length 25 received.
Notification of length 152 received.
Notification of length 152 received.
Notification of length 152 received.
Msg of length 4096 received from ::1:54819 on stream 0 with SSN 0 and TSN 2575394654, PPID 0, context 47, complete 1.
Msg of length 4096 received from ::1:54819 on stream 0 with SSN 1 and TSN 2575394658, PPID 0, context 47, complete 1.
Notification of length 12 received.
Notification of length 20 received.
^C

[tuexen]

You need to enable the explicit EOR marking as described in Socket API.

[dirkz]

Thanks for pointing that out.

Enabling SCTP_EXPLICIT_EOR on both client and server, the client now gets "Connection reset by peer". Will read RFC 6458 to make sure I'm using it correctly, and then get back if needed.

usrsctp (arbitrary_size)$ ./Build/programs/discard_server 423 324

Notification of length 25 received.
Notification of length 152 received.
Notification of length 152 received.
Notification of length 152 received.
Notification of length 152 received.
Notification of length 28 received.

usrsctp (arbitrary_size)$ ./Build/programs/client 127.0.0.1 9 0 324 423
*** Sent 4096 bytes (262144 available), completed 0
handle_notification : SCTP_ASSOC_CHANGE
Association change SCTP_COMM_UP, streams (in/out) = (10/10), supports PR AUTH ASCONF MULTIBUF RE-CONFIG.
handle_notification : SCTP_PEER_ADDR_CHANGE
Peer address ::1 is now SCTP_ADDR_CONFIRMED (error = 0x00000000).
Local addresses: 169.254.177.108, 192.168.1.35, ::1, 127.0.0.1.
Peer addresses: 127.0.0.1, 169.254.177.108, 192.168.1.35, ::1.
*** Sent 4096 bytes (259492 available), completed 0
handle_notification : SCTP_PEER_ADDR_CHANGE
Peer address 169.254.177.108 is now SCTP_ADDR_CONFIRMED (error = 0x00000000).
handle_notification : SCTP_PEER_ADDR_CHANGE
Peer address 192.168.1.35 is now SCTP_ADDR_CONFIRMED (error = 0x00000000).
Could not send bytes: Connection reset by peer
Could not send bytes: Connection reset by peer
Could not send bytes: Connection reset by peer
Could not send bytes: Connection reset by peer
handle_notification : SCTP_SEND_FAILED_EVENT
Unsent message with PPID = 0, SID = 0, flags: 0x0001 due to error = 0x0000000c 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00.
handle_notification : SCTP_ASSOC_CHANGE
Association change SCTP_COMM_LOST, streams (in/out) = (10/10).

[lgrahl]

Note that handling MSG_EOR on the receiver side is mandatory while explicitly marking a message as complete when sending with SCTP_EXPLICIT_EOR and SCTP_EOR is optional. Both are API constructs and while they do very similar things they are not orthogonal (e.g. you may get differently sized chunks on the receiver side to the one from the sender side).

[dirkz]

In my sample (client.c), I enable explicit EOR marking. send_cb gets called on connect with 262144 bytes, 256K, available for sending. I send 128K with MSG_EOR.

After that the client receives SCTP_ASSOC_CHANGE with SCTP_COMM_UP.

After one or two SCTP_PEER_ADDR_CHANGE the client receives SCTP_SEND_FAILED_EVENT.

In the end, the server (discard_server.c) only receives 66432 bytes (from 128K sent) without MSG_EOR.

What am I doing wrong?

[lgrahl]

Do the default usrsctp examples work?

[tuexen]

ekr_loop in particular.

[dirkz]

Thanks for the feedback.

The examples work correctly. ekr_loop, and client with server. But client uses a tiny buffer of 80 bytes, and no EOR.

I will try to pinpoint my problem(s) based on client and server. First by introducing EOR handling in client (merely displaying it is enough for the server for this example), then increasing the buffer size, then maybe switching to send_cb, if needed.

If I use usrsctp_sendv for blocking transfer of big messages with EOR, since I never know how many bytes can be actually transferred, is it possible to send an empty message with just MSG_EOR? Tried that before, but it gave me send errors. If not, how is it possible to use MSG_EOR with usrsctp_sendv? Does the lib remove the MSG_EOR in case only part of the buffer can be transferred?

[tuexen]

Thanks for the feedback.

The examples work correctly. ekr_loop, and client with server. But client uses a tiny buffer of 80 bytes, and no EOR.

ekrloop was the one I was interested in.

I will try to pinpoint my problem(s) based on client and server. First by introducing EOR handling in client (merely displaying it is enough for the server for this example), then increasing the buffer size, then maybe switching to send_cb, if needed.

If I use usrsctp_sendv for blocking transfer of big messages with EOR, since I never know how many bytes can be actually transferred, is it possible to send an empty message with just MSG_EOR? Tried that before,

No, empty messages are not possible. The end of RFC 4960, Section 6.2 reads:

If an endpoint receives a DATA chunk with no user data (i.e., the
Length field is set to 16), it MUST send an ABORT with error cause
set to "No User Data".

An endpoint SHOULD NOT send a DATA chunk with no user data part.

but it gave me send errors. If not, how is it possible to use MSG_EOR with usrsctp_sendv? Does the lib remove the MSG_EOR in case only part of the buffer can be transferred?

That should be the case. If not, it is a bug.

[lgrahl]

Caution: SCTP_EOR (for sending) and MSG_EOR (for receiving) are not the same. Your code looks like it mixes those two up. See https://github.com/rawrtc/rawrtc-data-channel/blob/master/src/sctp_transport/transport.c#L2839-L2940 for an example on how to use it.

[dirkz]

Indeed, I mistakenly assumed it's MSG_EOR on both client and server. Using SCTP_EOR fixes my remaining issues. Plus, a client-set SCTP_EOR is gracefully handled if only parts can be sent.

Recap how to solve this issue, with the goal to send arbitrarily sized messages, using only sync/blocking calls as a bonus:

• Enable SCTP_EXPLICIT_EOR with usrsctp_setsockopt on both client and server.
• Use struct sctp_sndinfo on the client, setting enabling SCTP_EOR in sctp_sndinfo.snd_flags as needed.
• Checking the flags (from usrsctp_recvv) for a set MSG_EOR on the server.

client.c
server

Thank you very much for your help. This issue is solved from my side.