An endpoint is "force detached" by the broker when sending messages

[adamralph]

Describe the bug

Description

When sending/publishing messages (either using transactional session or inside a message handler), the default behaviour is to batch them and send them after the method calling the send/publish method has exited. (This behavior can be changed to dispatch immediately, but then transactionality is lost.)

Expected behavior

The messages are sent.

Actual behavior

The messages are not sent, and an exception is logged.

Versions

3.0.0 onwards. Introduced in #644

Steps to reproduce

In a message handler, send some messages, with the total size of all messages greater than the max allowed message size of the queue.

Relevant log output

> System.InvalidOperationException: The link 'G6:14740510:amqps://**namespace removed**.servicebus.windows.net/-dd910ad4;0:7:8' is force detached by the broker because publisher(link3082) received a batch message with no data in it. Detach origin: Publisher.
For troubleshooting information, see https://aka.ms/azsdk/net/servicebus/exceptions/troubleshoot.

Additional Information

Workarounds

None identified so far.

Note that switching to immediate dispatch (actually sending the message when the send/publish method is called) is not a workaround, because that is very different behaviour which loses any guarantees around message sending/publishing transactionality.

Possible solutions

• Measure the size of the outgoing batch before sending it, and if it's too large, don't batch.
• Catch the exception, and retry in a non-batched manner. This probably requires some better behaviour from the Azure SDK client, see [BUG] Batch.TryAdd returns true on premium with large message support but sending the batch throws an InvalidOperationException Azure/azure-sdk-for-net#44261
• Do not fix the bug, but instead provide a configuration switch to switch off batching. This may not be sufficient to fix the bug in ServiceControl (see below).

Additional information

This also manifests in ServiceControl when it attempts to forward audit messages. See Particular/ServiceControl#4185.

Our resolution to this bug may well depend on changes to the Azure SDK client, see Azure/azure-sdk-for-net#44261

[adamralph]

We've prioritized this near the top of our list and will work on it as soon as we can.