[BUG] Many abandoned messages in queues/topics

[Alex-Boo]

Description
Recently we migrated from WindowsAzure.ServiceBus to Azure.Messaging.ServiceBus. After deployment to production we noticed that processing of messages is slow down and there are many abandoned messages in queue/topics.

There is screenshot of metrics before deployment (WindowsAzure.ServiceBus):

And there is screenshot of current state (Azure.Messaging.ServiceBus):

We investigated the issue and found:

1. We do not abandon messages in code, it happens inside of Azure.Messaging.ServiceBus
2. Topics and queues are affected
3. We have the issue only if ServiceBusReceiverOptions.PrefetchCount is not defined.

Expected behavior
No abandoned messages if we do not abandon them directly in code.

Actual behavior (include Exception or Stack Trace)
Lot's of abandoned massages came from the library.

To Reproduce
I was able to reproduce the problem in console application:

	class Program
	{
		static void Main(string[] args)
		{
			var receiver = new Receiver4();
			receiver.ReadMessages().GetAwaiter().GetResult();			
		}
	}

    class Receiver4
    {
        private const string ConnectionString = "connection string";

        public async Task ReadMessages()
        {
            var sbOptions = new ServiceBusClientOptions()
            {
                RetryOptions = new ServiceBusRetryOptions
                {
                    Mode = ServiceBusRetryMode.Fixed,
                    Delay = TimeSpan.FromSeconds(1),
                    MaxDelay = TimeSpan.FromSeconds(5),
                    MaxRetries = 1,
                    TryTimeout = TimeSpan.FromSeconds(30)
                }
            };

            ServiceBusClient client = new ServiceBusClient(ConnectionString, sbOptions);

            ServiceBusReceiverOptions options = new ServiceBusReceiverOptions
            {
            };

            var receiver = client.CreateReceiver("alex-test", options);

            var receivedMessages = new List<ServiceBusReceivedMessage>();
            while (true)
            {
                Console.WriteLine("Reading");

                var messages = await receiver.ReceiveMessagesAsync(128, TimeSpan.FromSeconds(5)).ConfigureAwait(false);

                Console.WriteLine($"{messages.Count} were received");

                foreach (var message in messages)
                {
                    Console.WriteLine(message.Body.ToString());

                    receivedMessages.Add(message);
                }

                await Task.Delay(100);

                Console.WriteLine("Completing");

                foreach (var message in receivedMessages)
                {
                    try
                    {
                        await receiver.CompleteMessageAsync(message).ConfigureAwait(false);
                    }
                    catch (ServiceBusException ex) when (ex.Reason == ServiceBusFailureReason.MessageLockLost)
                    {
                        Console.WriteLine("MessageLockLost, {0}", message.Body.ToString());
                    }
                }

                receivedMessages.Clear();

                Console.WriteLine("Completed");

                ConsoleKeyInfo key = Console.ReadKey();

                if (key.KeyChar == 's')
                {
                    break;
                }
            }
        }
    }

Environment:

• Name and version of the Library package used: Azure.Messaging.ServiceBus 7.3.0
• Hosting platform or OS and .NET runtime version Azure Cloud Services
• IDE and version : Visual Studio 16.11.4

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[JoshLove-msft]

Hi @Alex-Boo,
The SDK won't abandon messages for you when using the ServiceBusReceiver. It seems that the lock is being lost for some of the messages. Are you using the default 30 second lock duration for your queue? If so, I wouldn't expect the lock to be lost when processing 128 messages. Can you include the output from your test console app?

[Alex-Boo]

Hello,
Message lock duration is 5 minutes for queue that I use in the console app and 2 minutes for queues/topics in production.

I added timestamp to output to make it more clear:

and there is a screenshot from Azure Portal: