Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Messages ordering when receiving batch of messages from Service Bus #43026

Closed
junaidahmed92 opened this issue Mar 27, 2024 · 13 comments
Closed
Assignees
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. issue-addressed The Azure SDK team member assisting with this issue believes it to be addressed and ready to close. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Bus

Comments

@junaidahmed92
Copy link

Library name and version

Azure.Messaging.ServiceBus 7.15.0

Query/Question

I'm receiving messages in batch from Azure Service Bus:

IReadOnlyList<ServiceBusReceivedMessage> receivedMessages = await serviceBusReceiver.ReceiveMessagesAsync(maxMessages: maxMessages, maxWaitTime: TimeSpan.FromSeconds(1), cancellationToken: cancellationToken);

It works fine but I am unsure if it also consider the ordering (FIFO). I know what we can achieve this behaviour using Sessions. But I want to ask if I need to use sessions when receiving batch of messages?

I've a single receiver that receives and process messages in batches. But I want to ensure ordering as I am receiving devices info e.g last seen etc and I need to make sure that I've updated the correct info.

Also if Sessions is the only way then should I just set the SessionId to DeviceId and that will make sure that the all the messages linked to a single device get processed in order?

I've done a quick test and I can see that message are received in sequence. I've added batch of 150 messages to the topic:

int numOfMessages = 150;
for (int i = 1; i <= numOfMessages; i++)
{
    // try adding a message to the batch
    var msg = new ServiceBusMessage($"Message {i}");
    msg.ApplicationProperties.Add("Type", 1);
    if (!messageBatch.TryAddMessage(msg))
    {
        // if it is too large for the batch
        throw new Exception($"The message {i} is too large to fit in the batch.");
    }
}

try
{
    
    await sender.SendMessagesAsync(messageBatch);
    Console.WriteLine($"A batch of {numOfMessages} messages has been published.");
    
}
finally
{
    // Calling DisposeAsync on client types is required to ensure that network
    // resources and other unmanaged objects are properly cleaned up.
    await sender.DisposeAsync();
    await client.DisposeAsync();
}

After this I added a receiver method that receives and logs the message to the console:

private async Task ProcessMessages(int maxMessagesToProcess, CancellationToken cancellationToken)
{
    
     while (!cancellationToken.IsCancellationRequested)
     {
         var receivedMessageList = new List<ServiceBusReceivedMessage>();
         IReadOnlyList<ServiceBusReceivedMessage> receivedMessages = await _serviceBusReceiver.ReceiveMessagesAsync(maxMessages: maxMessagesToProcess, maxWaitTime: TimeSpan.FromSeconds(1), cancellationToken: cancellationToken);
         if (receivedMessages.Count > 0)
         {
             receivedMessageList.AddRange(receivedMessages);
             foreach (var msg in receivedMessageList)
             {
                 Debug.WriteLine($"Message: {msg.Body}");
             }
             if (receivedMessageList.Count >= ConcurrentMessagesToProcess)
             {
                 //await ProcessReceivedMessages(receivedMessageList);

                 var processAgain = await CheckForMessageProcessing();
                 if (!processAgain)
                 {
                     break;
                 }
             }
         }
         else
         {
             break;
         }
     }
 }

After multiple tests, I get the messages in sequence. I'm unsure if I've missed any scenario or is this expected.

Environment

.Net Core 3.1
Visual Studio 2022

@github-actions github-actions bot added Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Bus labels Mar 27, 2024
Copy link

Thank you for your feedback. Tagging and routing to the team member best able to assist.

@jsquire jsquire assigned jsquire and unassigned JoshLove-msft Mar 27, 2024
@jsquire
Copy link
Member

jsquire commented Mar 27, 2024

Hi @junaidahmed92. Thanks for reaching out and we regret that you're experiencing difficulties. The client will always return messages to you in the order that the Service Bus returns them. What you need to be aware of, however, is that the Service Bus service itself does not guarantee message ordering unless sessions are used.

Assuming no errors and normal operation, you'll usually see messages streamed to the client in the order that they were received by the service. However, messages which are locked by a reader are not sent by the service and, thus, may appear out-of-order if the holder of the lock does not complete the message. For example, Reader A and Reader B are both reading from a queue. Only one reader has control of a message at any time. Reader A will not see the messages locked by Reader B and vice-versa. However, if Reader B abandons a message that it has read, that message stays in the queue any may be read by Reader A who would see it out-of-order.

This is also true if your application has the message, but the lock expires before you complete it. For example, Reader A gets messages 1, 2, 3, 4 from the service. Processing takes a while and the message lock expires before message 4 is done processing. When Reader A goes to complete message 4, the service will not allow it. The next time Reader A reads, it will see message 4 again, out-of-order.

As for choosing a session identifier, anything that is unique and can be used to group related messages together would work. More information can be found in First-in, first-out (FIFO) pattern.

@jsquire jsquire added the issue-addressed The Azure SDK team member assisting with this issue believes it to be addressed and ready to close. label Mar 27, 2024
Copy link

Hi @junaidahmed92. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

@github-actions github-actions bot removed the needs-team-attention This issue needs attention from Azure service team or SDK team label Mar 27, 2024
@junaidahmed92
Copy link
Author

Thanks @jsquire for tbe detailed answer.
So this means if i add a same sessionid to all messages that are picked by a specific topic subscription, i will receive them in order.
E.g I've a Device-subscription and I add a sessionid "device-session" to all the messages.

@jsquire
Copy link
Member

jsquire commented Mar 27, 2024

Thanks @jsquire for tbe detailed answer. So this means if i add a same sessionid to all messages that are picked by a specific topic subscription, i will receive them in order. E.g I've a Device-subscription and I add a sessionid "device-session" to all the messages.

Correct, assuming that you're not receiving/processing in parallel and prefetch is not enabled.

@junaidahmed92
Copy link
Author

junaidahmed92 commented Mar 27, 2024

No, I'm not processing in parallel. I am fetching a batch of messages and processing them in loop.
However prefetch is enabled on the ServiceBusSessionReceiver. Should I disable it?

@jsquire
Copy link
Member

jsquire commented Mar 27, 2024

However prefetch is enabled on the ServiceBusSessionReceiver Should I disable it?

If strict ordering guarantees are needed, yes. There's a known issue in the AMQP transport library that we use which causes prefetch + sessions to ignore ordering in a rare edge case. The team that owns the transport is investigating, but the behavior has only been observed in one application and is non-deterministic so they're still attempting to find the root cause.

@junaidahmed92
Copy link
Author

Thanks @jsquire for the clarification. I've changed my implementation. However I've got a question, what if I've only receiver who is responsible to receive all messages and if there's an error or a lockdown expires, the same receiver will receive and process it again. Are using sessions still preferable in this scenario?

@junaidahmed92
Copy link
Author

junaidahmed92 commented Mar 28, 2024

I've got couple of more questions:

  1. I've set a peek lock time for 5 mins and my SessionReceiver's receivemode is set to PeekLock but even after a min I can fetch the same message again. This could raise a problem if I've multiple listeners and one is already processing a message.
ServiceBusSessionReceiver receiver = await _serviceBusClient.AcceptNextSessionAsync(topicName, subscriptionName, new ServiceBusSessionReceiverOptions
{
    ReceiveMode = ServiceBusReceiveMode.PeekLock
});
  1. The other question is related to the delivery count. I have noticed that even if the message processing throws exception, the delivery count always remains 0 and based on the this doc, it's expected. But the problem is, if a single message in a session is unable to get processed, it will keep on retrying forever and will never goes to the DLQ. What are the options to tackle this? I've set a DeliveryCount to 5 and I want to throw the session message to DLQ after 5 attempts. But if on retry the DeliveryCount is always 1 if I attach a breakpoint.

@jsquire
Copy link
Member

jsquire commented Mar 28, 2024

  1. If the session lock is held, you cannot read a message for the same session with two receivers. If you abandon the message but still hold the session lock, that message is available to be read immediately but only by the receiver holding the session lock. There are no individual message locks when using sessions - the session itself is the lock.

  2. Delivery count is a broker-owned concept that the client library has no insight into nor influence over. The Azure SDK team cannot speak to the behavior here authoritatively. One thing that I can mention is that you do have the ability to manually dead-letter a message and do not need to rely on the broker to do so via delivery count semantics.

@junaidahmed92
Copy link
Author

Thanks for the clarification. But when I wss not using sessions the delivery count was completely fine. With ServiceBusReader i mean.
I didn't have to manually retry and DLQ the messages but rather service bus was doing that.

Also can you confirm if I only have a single ServiceBusReceiver, is tbe messaging ordering still a problem without sessions.

@jsquire
Copy link
Member

jsquire commented Mar 28, 2024

Thanks for the clarification. But when I wss not using sessions the delivery count was completely fine. With ServiceBusReader i mean.
I didn't have to manually retry and DLQ the messages but rather service bus was doing that.

Delivery count is a broker-owned field and actions taken are by the service; the Azure SDK clients don't play a part in how it works - sessions or no. It's all service behavior, so there's nothing that I can offer beyond what I've already shared.

Also can you confirm if I only have a single ServiceBusReceiver, is tbe messaging ordering still a problem without sessions.

Yes, for the reasons that I mentioned above. Any scenario where the lock expires, you have an exception in your processing, your application or its host crashes/reboots/migrates can cause messages to be delivered out of order. They're edge cases, but not uncommon ones. The salient point is that the Service Bus service makes no guarantees about strict ordering unless sessions are used.

Here, again, the client has no insight nor influence; we just return what the service gave us in the order that it gave it to us.

@junaidahmed92
Copy link
Author

Alright, thanks. Closing this issue now.

@github-actions github-actions bot locked and limited conversation to collaborators Jun 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. issue-addressed The Azure SDK team member assisting with this issue believes it to be addressed and ready to close. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Bus
Projects
None yet
Development

No branches or pull requests

3 participants