[BUG] Service Bus Processor get stuck when not completing or abandoning messages with prefetch count > 0

[Vandersteen]

Library name and version

Azure.Messaging.ServiceBus 7.18.4

Describe the bug

When the prefetch count is set to 0 and autocompletemessage is set to false, the code below works as expected.
The processor will pick up the next message and continue processing.
The message will eventually be unlocked and tried again later.

If we set the prefetch count > 0, this does not happen. The processor gets stuck and does not pick up new messages

var serviceBusClient = new ServiceBusClient("xxx");
var processor = serviceBusClient.CreateProcessor("topic", "subscription", new ServiceBusProcessorOptions()
{
    PrefetchCount = 1,
    AutoCompleteMessages = false
});

processor.ProcessMessageAsync += async args =>
{
    var success = false;
    if (success)
    {
        await args.CompleteMessageAsync(args.Message);
    }
    else
    {
        //We want the message to be retried when the lock expires
        //So we don't complete or abandon it
        Console.WriteLine($"Not completing message {args.Message.MessageId}");
    }
};

Expected behavior

The processor does not halt

Actual behavior

The processor is stuck and does no longer consume messages

Reproduction Steps

Set the prefetch count to > 0

Environment

.NET SDK:
Version: 8.0.404
Commit: 7b190310f2
Workload version: 8.0.400-manifests.2877bc3a
MSBuild version: 17.11.9+a69bbaaf5

Runtime Environment:
OS Name: Mac OS X
OS Version: 15.0
OS Platform: Darwin
RID: osx-x64

[github-actions]

Hello @Vandersteen. I'm an AI assistant for the azure-sdk-for-net repository. I have some suggestions that you can try out while the team gets back to you.

• Prefetching will retrieve messages into an internal buffer. When a message is not explicitly completed or abandoned, it remains in that buffer until the lock expires. This can block new messages from being fetched when PrefetchCount > 0.
• For scenarios where you rely on lock expiration for retry, consider either setting PrefetchCount to 0 or explicitly calling Abandon (or deferring) on messages that aren’t successfully processed.
• Review the effects of message prefetching in the context of manual message settlement. More details in the Service Bus documentation: https://learn.microsoft.com/dotnet/api/azure.messaging.servicebus.servicebusprocessoroptions.prefetchcount?view=azure-dotnet
• This behavior is by design to improve throughput, so adjusting the pattern or prefetch settings might be required to meet your expected processing flow.

The team will get back to you shortly, hopefully this helps in the meantime.

[jsquire]

Hi @Vandersteen. Thanks for reaching out and we regret that you're experiencing difficulties. We're unable to reproduce the behavior that you're seeing with the available context. In both our nightly test runs and in a stand-alone app, I'm seeing the configuration that you've shared work as expected.

In order to assist, we'll need to ask for some additional information. Are you able to reproduce this in a small, stand-alone console application along with an ARM or bicep template that you can share? It would also be helpful to capture a +/- 5-minute slice of Azure SDK logs at the verbose level around the behavior that you're seeing and share them so that we can understand the processor's view of the environment.

[github-actions]

Hi @Vandersteen. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[Vandersteen]

Hi @jsquire you can find a repro here https://github.com/Vandersteen/ServiceBusRepro

The README.md contains the steps to set everything up. You will only need a 'standard tier' SB and a connectionstring which also has admin rights to set up the topic & subscription