[Service Bus] The code always throws error "Operation to settle the message has timed out. The disposition of the message may or may not be successful" after processing message.

[dimalama]

• @azure/service-bus:
• 1.0.2:
• Mac OS:
• nodejs
  • 11.2.0:
• typescript
  • 3.5.1:

The error "Operation to settle the message has timed out. The disposition of the message may or may not be successful" happens every time during message processing
I have created topic inside AzurePortal => Service Bus and created subscription to that topic.
I copied the example from documentation.

const { ServiceBusClient, ReceiveMode, delay } = require('@azure/service-bus');

export class SubscriptionService {
  constructor() {
    this.main().catch((err) => {
      console.log('Error occurred: ', err);
    });
  private async main() {
   const sbClient = ServiceBusClient.createFromConnectionString(process.env.AZURE_SERVICEBUS_CONNECTION_STRING || '');

    // If receiving from a Subscription, use `createSubscriptionClient` instead of `createQueueClient`
    const queueClient = sbClient.createSubscriptionClient(TestTopic, 'test');

    // To receive messages from sessions, use getSessionReceiver instead of getReceiver or look at
    // the sample in sessions.js file
    const receiver = queueClient.createReceiver(ReceiveMode.peekLock);

    const onMessageHandler = async (brokeredMessage: any) => {
      console.log(`Received message: ${brokeredMessage.body}`);
      await brokeredMessage.complete();
    };
    const onErrorHandler = (err: any) => {
      console.log("Error occurred: ", err);
    };

    try {
      receiver.registerMessageHandler(onMessageHandler, onErrorHandler, { autoComplete: false });

      // Waiting long enough before closing the receiver to receive messages
      await delay(5000);

      await receiver.close();
      await queueClient.close();
    } finally {
      await sbClient.close();
    }
  }
}

onMessageHandler receives a message from the topic. The code processes the message. Message got removed from Azure Portal => Service Bus. I get the error Operation to settle the message has timed out. The disposition of the message may or may not be successful inside onErrorHandler after onMessageHandler is done executing. Error happens every time.

To Reproduce
Steps to reproduce the behavior:

1. Create topic in service bus
2. Create subscription to the topic
3. Use the code from documentation

const { ServiceBusClient, ReceiveMode, delay } = require("@azure/service-bus");

// Define connection string and related Service Bus entity names here
const connectionString = "";
const queueName = "";

async function main() {
  const sbClient = ServiceBusClient.createFromConnectionString(connectionString);

  // If receiving from a Subscription, use `createSubscriptionClient` instead of `createQueueClient`
  const queueClient = sbClient.createQueueClient(queueName);

  // To receive messages from sessions, use getSessionReceiver instead of getReceiver or look at
  // the sample in sessions.js file
  const receiver = queueClient.createReceiver(ReceiveMode.peekLock);

  const onMessageHandler = async (brokeredMessage) => {
    console.log(`Received message: ${brokeredMessage.body}`);
    await brokeredMessage.complete();
  };
  const onErrorHandler = (err) => {
    console.log("Error occurred: ", err);
  };

  try {
    receiver.registerMessageHandler(onMessageHandler, onErrorHandler, { autoComplete: false });

    // Waiting long enough before closing the receiver to receive messages
    await delay(5000);

    await receiver.close();
    await queueClient.close();
  } finally {
    await sbClient.close();
  }
}

main().catch((err) => {
  console.log("Error occurred: ", err);
});

4. Execute code

Expected behavior
Process and clear up messages inside the topic without error.

Screenshots
Receive message from topic inside onMessageHandler:

When onMessageHandler is done executing:

Error:

Additional context
Full error:

condition:"com.microsoft:timeout"
info:undefined
message:"Operation to settle the message has timed out. The disposition of the message may or may not be successful"
name:"ServiceUnavailableError"
retryable:true
stack:"ServiceUnavailableError: Operation to settle the message has timed out. The disposition of the message may or may not be successful\n    at Object.translate (/Users/d/Projects/SB/node_modules/@azure/amqp-common/dist/index.js:741:17)\n    at Timeout.setTimeout [as _onTimeout] (/Users/d/Projects/SB/node_modules/@azure/service-bus/dist/index.js:2317:46)\n    at listOnTimeout (timers.js:324:15)\n    at processTimers (timers.js:268:5)"
translated:true

[ramya-rao-a]

Thanks for reporting @dlukianenko

When a request to settle a message (in your case "complete" a message) is made to Service Bus, the library waits for 20 seconds to hear back an acknowledgement that the message has indeed been successfully settled. The error you are seeing is thrown when this 20 second duration times out.

@binzywu What should be the ideal timeout duration for this case i.e. how long should the client library wait to hear back an acknowledgement from the service? Can you think of any reason why the service would fail to respond within the 20 sec timeframe?

[jfggdl]

@dlukianenko, would you please create a support request from the Azure Portal for this issue and include in it more contextual information like the name of your namespace? This is an issue for which we need to do further investigation on the service side. 20 seconds timeout should be enough and the fact that you are getting an error may be indicative of some other problem condition.

I am going to close this issue, but feel free to reopen it if you have other kind of questions.

[jfggdl]

@dlukianenko

[jfggdl]

@dlukianenko, did you create the support request? Would you please share the SR#?

[dimalama]

Hello, I'm on a dev-test subscription and currently, I don't have a tech support subscription, so I cannot really create actual ServiceBus support requests without paying for tech support subscription. Is there any other option available to me?

[SamJarmanPP]

We've also seen this over 1500 times since our service using service into production a few months ago. Are there any queue configurations to check or anything we can do before raising a support query? Thanks @ramya-rao-a :)

[SamJarmanPP]

As a work around, would it make sense to check message.isSettled before processing? Does that get successfully written in the case of these timeouts?

[ramya-rao-a]

@SamJarmanPP Unfortunately no. he isSettled property on a message gets set on the service side before it sends the message to the client. Once the client makes the request to settle the message, the client then waits for an acknowledgement from the service. This is where the time out happens. The message object is entirely on the client side at this point and is not mutated.

[SamJarmanPP]

@ramya-rao-a Ah okay, darn. I asked over here what we should do in these cases - looking forward to hearing back :) We just want to make sure the message is processed once and only once, and that this is guaranteed. (Are we asking too much? haha)

[binzywu]

@SamJarmanPP please could you open a support case and provide us more details of namespace, entity, timestamp and ideally few sample message ids will be very helpful.

[SamJarmanPP]

@binzywu Sure but I feel the answer you provide would be applicable to anyone receiving this error?

[ramya-rao-a]

@SamJarmanPP Opening a support case/ticket will help @binzywu's team investigate the problem because they can then check the logs from the service side as well. Once they come to a conclusion on what the problem is, @binzywu or his team will surely update this GitHub issue with the answer and possible steps that can be taken to fix it

[SamJarmanPP]

@binzywu @ramya-rao-a opened one yesterday - 119081423002394 :)

[axisc]

Thanks @SamJarmanPP,

I'll close the issue now and we can track this via the support ticket.

[ramya-rao-a]

@axayjo Based on the support ticket logged by @SamJarmanPP, his issue is with what is being discussed at #2828 (comment)

Originally, this issue was logged by @dlukianenko who doesn't have a subscription with tech support. Can you suggest any other way you can look into why the service is not able to settle messages within 20 seconds?

[axisc]

@dlukianenko

We will need to access the service logs for the specified namespace to debug the root cause of the delay in completing the message.

In the meantime, there are some options

• Increasing the operation timeout (in other lang clients, the default timeout is 60 seconds)
• Rely on retry logic

I'm closing the issue for now. Please feel free to reopen if you still have questions.

[ramya-rao-a]

The latest version of the Service Bus library (1.0.4) has the timeout increased from 20 seconds to 60 seconds as suggested by @axisc

It also has the fix for the issue reported by @SamJarmanPP where messages failed to settle repeatedly after the library recovered from a broken receiver link