Preserve order of broker messages [SPR-13989]

[spring-projects-issues]

Pieter opened SPR-13989 and commented

M1 is a large message and M2 is a small message.
M2 is sent immediately after M1.
=> M2 is received before M1 although the messageId of M1 is smaller than the messageId of M2

I enabled tracing for 'org.springframework.web.socket':
clientOutboundChannel-90 processes M1
clientOutboundChannel-91 processes M2

10:43:01 [clientOutboundChannel-91] WebSocketServerSockJsSession - Cancelling heartbeat in session qdivbsu4
10:43:01 [clientOutboundChannel-91] WebSocketServerSockJsSession - Preparing to write SockJsFrame content='a["MESSAGE\ndestination:/user/queue/reply-1d07e5d5-91f1-4617-9f8e-477041972ad5\n...(truncated)'
10:43:01 [clientOutboundChannel-91] WebSocketServerSockJsSession - Writing SockJsFrame content='a["MESSAGE\ndestination:/user/queue/reply-1d07e5d5-91f1-4617-9f8e-477041972ad5\n...(truncated)'
10:43:01 [clientOutboundChannel-91] NativeWebSocketSession - Sending TextMessage payload=[a["MESSAGE..], byteCount=14989, last=true], JettyWebSocketSession[id=6baad94d, uri=ws://localhost:8080/weaver/ep/307/qdivbsu4/websocket]
10:43:01 [clientOutboundChannel-91] WebSocketServerSockJsSession - Scheduled heartbeat in session qdivbsu4
10:43:01 [clientOutboundChannel-90] WebSocketServerSockJsSession - Cancelling heartbeat in session qdivbsu4
10:43:01 [clientOutboundChannel-90] WebSocketServerSockJsSession - Preparing to write SockJsFrame content='a["MESSAGE\ndestination:/user/queue/reply-1d07e5d5-91f1-4617-9f8e-477041972ad5\n...(truncated)'
10:43:01 [clientOutboundChannel-90] WebSocketServerSockJsSession - Writing SockJsFrame content='a["MESSAGE\ndestination:/user/queue/reply-1d07e5d5-91f1-4617-9f8e-477041972ad5\n...(truncated)'
10:43:01 [clientOutboundChannel-90] NativeWebSocketSession - Sending TextMessage payload=[a["MESSAGE..], byteCount=4556671, last=true], JettyWebSocketSession[id=6baad94d, uri=ws://localhost:8080/weaver/ep/307/qdivbsu4/websocket]
10:43:01 [clientOutboundChannel-90] WebSocketServerSockJsSession - Scheduled heartbeat in session qdivbsu4

I suspect that because the conversion of the payload of the smaller message is so fast that it passes the conversion of the larger payload and hence is sent before the larger message.

---

Affects: 4.2.5

Issue Links:

• Preserve-publish-order is mentioned in documentation of 5.0.x [SPR-17106] #21643 Preserve-publish-order is mentioned in documentation of 5.0.x

1 votes, 5 watchers

[spring-projects-issues]

Rossen Stoyanchev commented

Can you clarify what's sending the message? Is this using a messaging template? Is it from the same or different threads? Are you using the simple broker or the broker relay?

[spring-projects-issues]

Pieter commented

All messages are sent from a single thread
I use an autowrired SimpMessageSendingOperations to send the messages
I didn't configure any broker, so I assume I use the simple broker

[spring-projects-issues]

Rossen Stoyanchev commented

Right, and I presume you haven't customized the brokerChannel? By default that sends the message (to the broker) in the same thread and the conversion also happens during that time so I don't think that has any impact. In other words the messages are likely getting processed by the broker sequentially (and logging could confirm that). More likely when the broker then sends the message to the clientOuboundChannel, which is backed by a thread pool, there is no guarantee they will be processed sequentially.

[spring-projects-issues]

Pieter commented

Indeed, I didn't customise the brokerChannel.
While debugging, I noticed that the messages were first sent on my application thread and later on again in the clientOutboundChannel.
The message receives a messageId in the clientOutboundChannel (and those are in the right order) en just before they are given to the ConcurrentWebSocketSessionDecorator the payload is first written to a memory stream and then converted to a String (UTF-8). I think that during these conversion operations the 2 messages change their order.

[spring-projects-issues]

Rossen Stoyanchev commented

In summary this is currently expected behavior. As messages pass through the clientOutboundChannel there is nothing to ensure they're sent in the same order.

[spring-projects-issues]

Pieter commented

It is a pity, that all clients that wants messages to arrive in the same order as the are sent, that they have to write and maintain code to achieve this.

If the final conversion of the STOMP message to a string happens in the ConcurrentWebSocketSessionDecorator in stead of in the clientOutboundChannel, then the chance that messages are reordered will be reduced.

[spring-projects-issues]

Rossen Stoyanchev commented

This is not in any way related to message conversion. The conversion happens immediately in the messaging template. You're sending them sequentially and the brokerChannel by default works in the thread of the sender. That means message M1 is fully converted and gets to the broker before the next message M2. The issue is that the thread pool backing the clientOutputboundChannel has no FIFO guarantees.

Strictly speaking we only need the thread pool for the actual sending of the message through the WebSocket session. In other words clientOutputboundChannel doesn't have to be backed by an executor strictly speaking. The StompSubProtocolHandler would then do its work preparing the WebSocketMessage (in the same thread but without blocking) and at that stage it's easier to submit to an executor while ensure the sequential order messages within a session.

[spring-projects-issues]

Rossen Stoyanchev commented

There is now a property in the Java config (MessageBrokerRegistry) and XML config (messsage-broker element), called "preservePublishOrder".

[spring-projects-issues]

Rossen Stoyanchev commented

This is also covered in the reference now.