Possible message loss with TimeToBeReceived on transactional MSMQ endpoints.

[janovesk]

We have discovered a bug in the TimeToBeReceived feature, which allows messages to automatically expire if not received quickly enough. If you use this in combination with the MsmqTransport and have transactions turned on, your messages might get the incorrect TimeToBeReceived value. This can lead to effective message loss when messages get removed from the receiving queue prematurely. It can also lead to an unintended build-up of messages in the receiving queue.

Unfortunately, we have no other choice than to disable this piece of functionality when running the MsmqTransport because of underlying issues in MSMQ. If we detect that you are sending messages with a TimeToBeReceived set on an endpoint with transactional MSMQ, we will throw an exception at runtime to remove any possibility of message loss. Further, we will throw an exception at startup if we detect that you have defined TimeToBeReceivedOnForwardedMessages in your UnicastBusConfig config section or OverrideTimeToBeReceived in your AuditConfig config section.

See the original issues, #2886 and #2941, for more details.

Who's affected

This affects you if you are sending messages with a non-default TimeToBeReceived from transactional endpoints that use the MsmqTransport. Please note that if you are running with the auditing or forwarding feature turned on, these messages are sent in the same transaction as your business messages. They are therefore also affected by this bug.

This also affects you if you have TimeToBeReceivedOnForwardedMessages in your UnicastBusConfig config section or a ForwardReceivedMessagesTo in your AuditConfig config section for a transactional endpoint that uses the MsmqTransport.

Symptoms

Messages sent do not get the configured TimeToBeReceived. All the messages in a transaction get the same TimeToBeReceived, even though you are sending messages of different types and with different TimeToBeReceived values. Messages are removed from the receiving queue prematurely or are not removed in a timely fashion.

What to do if you are affected

To avoid the runtime exception you have to stop using TimeToBeReceived on your messages. Make sure that none of the message types scanned by NServiceBus has a TimeToBeReceived attribute on them, and there are no TimeToBeReceived conventions defined.

You will also have to remove TimeToBeReceivedOnForwardedMessages from your UnicastBusConfig config section and OverrideTimeToBeReceivedfrom your AuditConfig config section to avoid having the endpoint refuse to start.

Workarounds

OverrideTimeToBeReceived/OverrideTimeToBeReceived is commonly used to make sure that your audit queue does not grow unbounded. You can no longer use this mechanism on the MsmqTransport. You will need to put in place a different mechanism to make sure the audit/forward queue does not fill up. One option is to use ServiceControl to consume the audit queue and lean on its built-in feature for auto-removal of old data. Another option is to schedule a script to periodically clean old messages from the queues yourself. Check out the details on how TimeToBeReceived works on the different transports.