Adding a Proper Outbox Feature

[GurGaller]

I think adding a persistent message outbox/inbox feature to MassTransit is a great idea.
If you're unfamiliar with the concept, I suggest you read this page from NServiceBus's documentation (they call it an Outbox, but it is actually an Inbox too. I think both patterns are valuable and we need both in MassTransit).

I know I'm not the first to ask for this, but I'm unsatisfied with the reasons presented for not adding it as a built-in feature.
I'll try to present my view on this, and why it is such an essential feature.

The way I see it, MassTransit is an abstraction on top of messaging infrastructure. There are two main justifications for MassTransit's existence:

1. It decouples applications from underlying messaging systems, thus allowing them to be replaced. This improves testability (we can swap out the slow network-bound implementation with a fake in-memory one), and it reduces lock-in to specific technologies/services (allowing teams to change their technology decision later on).
2. It provides an easy-to-use API on top of the lower-level SDKs provided by the messaging systems. Developers are free to focus on the specific application domain they are working on, and not deal with ACKs, retries, serialization, topology management, and so on, on their own.

The first point is nice, but replacing the messaging transport is not such a common task (apart from testing).

The second point is stronger, in my opinion, so let's focus on that for a moment:
We want to help developers focus on solving their application-specific problems, by moving as much generic burden as possible to the framework. However, there are many cross-cutting concerns applications need that are not included in MassTransit, and that's OK (for obvious reasons). For evaluating whether a feature belongs in the MassTransit framework or not, I think these are the main factors:

1. How common is the problem? How many of MassTransit's users face that problem?
2. How big (=time-consuming for application developers) is the problem?
3. How hard will it be to implement a generic built-in solution?
4. Is it MassTransit's place to do that? Does it belong in the set of general responsibilities defined for this particular framework? Or is it better to implement it as an independent component?

Let's put the Persistent Outbox feature to each of these tests, one by one:

1. MassTransit is mainly used for integrating components in distributed systems. Unless an application uses the in-memory transport, it has to deal with the complexity of achieving consistency in a distributed system. It is part of the very nature of such systems, and I won't say a lot more about it because everywhere you look on the internet you can see some article or talk about the CAP theorem:)
   So I argue that almost 100% of MassTransit's users face this issue (You build an application --> you have state, and you use MassTransit --> your application is distributed --> you need to deal with eventual consistency). Note that maybe not all users are aware of the challenges of managing state in distributed systems - but I hope they realize them before their production environments fail on Friday afternoon.
2. Assuming the application developers didn't implement these infrastructure components on their own, let's see how they would have to deal with this problem at the application level:

• Without the deduplication that an inbox provides, all message handlers will have to be idempotent, to allow for safe retries. This is not a trivial task, and even if they keep that in mind all the time, they may make mistakes. Mistakes that will be very hard to catch. They can't be found with static analysis, and integration tests for every case (to verify the side effects did not occur more than once) - is not a cheap option (and I don't think it is reasonable to expect people to do it).
• Without a persistent outbox, all message handlers that publish other messages must detect that a retry was made (and that the state was saved), then generate all the messages that would have been published if the operation would have succeeded in the first place, then publish these messages manually. Again, I don't think it is reasonable to expect application developers to deal with all this. It is not a trivial task. Most operations are not naturally idempotent so you'll have to implement domain-specific deduplication in each of your handlers.
  I seriously doubt that the majority of MassTransit users implement this correctly in all of their message handlers. And if they don't they would have even larger issues regarding data corruption. These are not minor issues. At best (when implemented properly) they are time-consuming repetitive tasks that could have been handled in a central location by the infrastructure, but they are also extremely easy to forget or to implement incorrectly - causing data consistency issues that you may find only after they happen in production (where infrastructure starts to fail sometimes and retries are performed).

3. Not that hard. NServiceBus did a pretty good job explaining how they implemented this mechanism, and there are many others that published guidance on the subject. It is, however, persistence-technology-specific. So I think we should implement most of it in a database-independent fashion (and we can), and expose a minimal interface for persistence providers to plug in. We can, of course, also offer implementation for common persistence technologies like SQL Server or MongoDB.
4. I think it is. As a messaging framework, MassTransit manages the lifecycle of message processing. It should therefore be the one responsible for buffering messages (like it already does in-memory), de-duplicating them, and so on. This problem is very specific to messaging, which is MassTransit's core problem domain.

To summarize: a persistent inbox/outbox feature will shift a lot of common responsibility from the application level to the infrastructure level, allowing MassTransit users to focus on the specific problem they have at hand, instead of generic technical issues we all face, by developing applications with the assumption of an (almost*) exactly-once message delivery guarantee.

So what are we waiting for? Why not add this as a built-in feature?
(I would personally be happy to help with the implementation by the way)

*This is not magic. When dealing with external resources that are not covered by the transaction, idempotency will still need to be considered. However, I believe the majority of handlers do operate over a DB that supports ACID transactions - so even if this does not eliminate the problem in 100% of the cases, it will help a lot for sure.

[phatboyg]

Thanks for the well written essay on how a persistent outbox (or inbox) might be usable in specific scenarios to deal with consistency. There have been several threads, and lengthy conversations on why a persistent solution is better, and I have yet to see it proven given that even with NSB, you can only use it in a message handler (consumer), and even with that stated, they clearly define in the expectations:

The outbox is expected to generate duplicate messages from time to time, especially if there is unreliable communication between the endpoint and the message broker.

From my perspective and understanding, the whole point of an outbox is to deal with unreliable communication between the endpoint and the message broker.

I'm happy to entertain the inclusion of such a feature, but honestly we've tried and urgency hasn't been high. It's very database-specific (you can't have a generalized solution, since each database has different characteristics and constraints) so it has to be built for each database. Sure, there can be shared idioms, but each database would have its own set of guarantees.

And let's be honest, it isn't really an outbox. It's a deduplication solution that uses extensive database transactions to provide a reasonable guarantee of idempotent operation. The fact that it stages messages in the database for delivery once the transaction is completed is an aspect of it.

Related Stuff:

The on-ramp, essentially a database transport that is store-and-forward to the actual transport, for truly transactional storage and delivery of messages (incomplete): #2234

Reliable messaging: #577
Same discussion, less eloquently described: #2365

[GurGaller]

@phatboyg I think you misunderstood the quote from NServiceBus's documentation. The mechanism may cause the message handler to be executed more than once, but because there are UNIQUE constraints defined for the message IDs, the transaction will fail in such scenarios - leaving the database in a consistent state (and not publishing outgoing messages).

The only thing that is not covered by this infrastructure is integration with external systems (that can't be added to the transaction or be integrated using the same messaging transport). When implementing these parts - retries will have to be considered, sure. But I argue that what you referred to as "specific scenarios" is actually most of the cases.

And why isn't it an Outbox? messages are buffered instead of delivered right away, which is how the pattern got its name.
Store-and-forward solves the same issue and the only aspect where it is better (as you mentioned) is the support for atomic update-and-publish outside of message handlers, like in HTTP request handlers. I'm not necessarily suggesting NServiceBus's simpler implementation (based on message-broker retries) is a better fit here, it is just the solution I'm most familiar with.

And yeah, there is a non-negligible amount of work needed to be done on a per-database basis (especially for databases that require rigid schema), but

1. It is better to implement it once-per-database-type, than once-per-message-handler.
2. The part I think belongs to MT's core is the orchestration of this whole message handling process. Distributing other NuGet packages for specific databases is fine (and MassTransit already works that way for the transport infrastructure, for example).

I was unaware of PR #2234, and I'm happy to see progress in this area. I haven't got the chance to really look into what is included in the scope of this work but one thing I would love to see is support for non-relational databases that supports ACID transactions (like MongoDB 4.0+)

[GurGaller]

Oh, and as for monitoring a couple more SQL tables - I don't really see a problem with that, especially not for an opt-in feature.
It's not like we need a complete DB server for that, the outbox is by definition supposed to live next to the database tables we already have.

[GurGaller]

I have given it more thought and I changed my mind about the importance of supporting this outside of a message consumption context.
While it is true that non-idempotent HTTP requests with side effects can always lead to an inconsistent state, supporting atomic write-and-publish in this context is beneficial, because unlike when consuming a message, we can't trust the client to retry the request if it fails. And if the client doesn't retry, we may have an inconsistent state within our internal system (changes were written to the DB and messages weren't published, or vice-versa).

The solution NSB has for that is to just not do it. If you want to change the database and publish a message, just publish the message and do the change in the message's consumer. I don't like this because we may want to use the state in the DB to also construct the HTTP response (like returning an error status code in case of a UNIQUE constraint violation), instead of always returning an empty 202 response.

So while it is not as important and supporting it inside of a message consumption context, I think we should support the other scenario as well.

[phatboyg]

I've always agreed that "don't do it" is the right approach. Limit conversations to two parties:

• HTTP and Broker
• HTTP and Database
• Broker and Database (consumer utilized or saga database)

Once you bring in a third-party, that is where inconsistency is introduced. Consensus among two is hard enough in a disconnected environment, more parties = more complexity. So systems should be more of a relay than a meeting.

Of course, the next question is...

What if HTTP was the transport?
What if the database was the transport?

Those are hypothetical, but consider systems without messaging in place – they're basically HTTP and a Database. And we all know how well that has worked out over the long term, and the myriad of solutions created to overcome all of the challenges introduced.

[GurGaller]

@phatboyg I disagree with that. It is true that consistency is a complicated issue in distributed systems, but if we can move that complexity to the generic infrastructure modules I say it is worth it.
There is nothing wrong with an HTTP request handler writing to the database and publishing an event for downstream components.

While messaging has many advantages over communication using HTTP, in practice it is not used for integration with external systems. At some point, you'll want to call your system from within a web browser (that can't communicate using AMQP) or something like that.

If it is possible technically, I can't see any reason to restrict HTTP handlers even more, compared to what application developers are allowed to put in message consumers.

[phatboyg]

I don't think I'm explaining things in a way that you understand, so I'll just leave it.

[GurGaller]

One thing I do miss in NSB's implementation is supporting multiple outboxes in the same endpoint, for cases where you interact with more than one database (not as part of the same request)

[phatboyg]

I have a work-in-progress branch using Entity Framework Core as an inbox/outbox for message consumers or sagas. It isn't anywhere complete yet, but it's a start in finding the seams to make it work reliably. I'll update again once I have something ready for review/testing.