Moving messages with MSMQ to the error queue with TimeToBeReceived causes the endpoint to shutdown

[danielmarbach]

Who's affected

• All users with MSMQ and having messages with TimeToBeReceived set

Symptoms

When a message arrives on an endpoint with a TimeToBeReceived set and that messages fails to be processed (i.e. because of SerializationException) the message cannot be moved to the error queue. The following exception is raised:

InvalidOperationException: Sending messages with a custom TimeToBeReceived is not supported on transactional MSMQ

Since this is a fatal error which triggers the CircuitBreaker the endpoint will shutdown after a while. Restarts won't help since the message will be tried to process again and the end resullt will be another endpoint shutdwon

Description

Originally raised here Particular/ServiceControl#624

When a message can't be properly deserialized we try to send it to the error queue. But if that message contains a TimeToBeReceived the following excpetion is raised internally:

2016-01-27 10:53:55.0534|26|Fatal|NServiceBus|Fault manager failed to process the failed message with id cfe47b9b-99e0-4533-94a3-cb1731c97a69\6724376
System.InvalidOperationException: Could not forward failed message to error queue. ---> System.Exception: Failed to send message to address: Particular.ServiceControl.Errors@ENLIGHTEMENT. Sending messages with a custom TimeToBeReceived is not supported on transactional MSMQ.
   at NServiceBus.Transports.Msmq.MsmqMessageSender.Send(TransportMessage message, SendOptions sendOptions) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Transports\Msmq\MsmqMessageSender.cs:line 41
   at NServiceBus.Faults.Forwarder.FaultManager.HandleProcessingAlwaysFailsForMessage(TransportMessage message, Exception e, Int32 numberOfRetries) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Faults\Forwarder\FaultManager.cs:line 76
   at NServiceBus.Faults.Forwarder.FaultManager.TryHandleFailure(Action failureHandlingAction) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Faults\Forwarder\FaultManager.cs:line 112
   --- End of inner exception stack trace ---
   at NServiceBus.Faults.Forwarder.FaultManager.TryHandleFailure(Action failureHandlingAction) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Faults\Forwarder\FaultManager.cs:line 124
   at NServiceBus.Unicast.Transport.FirstLevelRetries.TryInvokeFaultManager(TransportMessage message, Exception exception, Int32 numberOfAttempts) in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Unicast\Transport\FirstLevelRetries.cs:line 66

https://github.com/Particular/NServiceBus/blob/master/src/NServiceBus.Core/Transports/Msmq/MsmqMessageSender.cs#L41

This then ultimately leads to a ServiceControl shutdown at some point.

It seems that in V5 we don't remove the TimeToBeReceived when sending to the error queue.

https://github.com/Particular/NServiceBus/blob/master/src/NServiceBus.Core/Faults/Forwarder/FaultManager.cs#L57

When fixing the bug we assumed passing in a new SendOption is enough. But the FaultManager uses the TimeToBeReceived set on the TransportMessage. So everytime we hit this the endpoint will trigger critical error and shutdown the endpoint but at least we don't loose a message

/ cc @janovesk

[danielmarbach]

@Particular/nservicebus-maintainers can someone quickly glance over it. I had to backport manually the tests and the implementation from 5.1.9 to 5.0.11. The nasty thing is I had to use dynamic for the builder fake :(

[timbussmann]

can't we move that to line 61 and delete the duplicate at 64?

[danielmarbach]

@timbussmann fixed. The duplication was introduced by the way I tested and evolved the code. Good catch

[timbussmann]

@danielmarbach how about renaming the issue title to something readable like "Reset TimeToBeReceived before moving message to ErrorQueue"?

[danielmarbach]

@timbussmann that slipped in. Changed

[SeanFeldman]

Removed workaround section as it's not helping customers.

Original section was:

• Remove the TimeToBeReceived from the message definition
• For messages in the queue, move the messages out of the queue into the error queue and remove the TimeToBeReceived