Outbox and Inbox pattern in .NET

Ensures reliable network communication (eventual consistency) between services.

Documentation | Packages

⭐ - The star motivates me a lot!

🔥 Examples of network communication problems

• Lost message

Amount was taken from bank account and transfer was never executed.

• The same message was processed again

Transfer was ordered and amount was debited from bank account twice.

• Unavailable service

Transfer order attempt fails.

🌟 Benefits

• Easy To Use
  • Quick learning and low entry threshold
  • Configured by default to prevent overwhelm for beginners
  • Add message to outbox and then appears in inbox handler
• Eventual Consistency
  • Outbox sends a message at least once
  • Inbox deduplicates the message and processes exactly once
• Scalability & Fault Tolerance
  • Retry pattern with delay
  • Multiple instances of the same service (distributed processing)
• Highly Configurable & Extendable
  • Components are configurable via settings
  • Extendable with new transport and storage providers
• Reduce Latency & Increase Bandwidth
  • Hook processing (instead of interval processing)
  • Compression algorithms
  • Grouping messages by topic and type to transport message (better compression)
  • Adjusting optimal transport message size

🎬 Run Sample

Note

Docker is required.

DEMO-PL.mp4

Clone this repo and open TransactionalBox.sln via Visual Studio 2022. Set the TransactionalBox.Sample.WebApi as startup and then run. You should see the following view.

Project is at an early stage of life cycle, if you find some bug, let me know ☎️.

Have fun 😃!

🏰 Architecture

The transactional box consists of two basic components. The following diagrams show the basic flow (omits details). They are designed to provide a general understanding of how transactional box works.

Outbox

The outbox is responsible for adding messages to the storage and then adding at least once to the transport.

Inbox

The inbox is responsible for getting messages from the transport and adding them to the storage, and then processes these messages.

✨ Features

Actions

• Add a message to send to the outbox
• Add a message to publish to the outbox
• Get messages from outbox and add them to transport
• Get a message from transport and add them to the inbox
• Get a message from inbox and process it

Storage

• InMemory (Default)
• Entity Framework (Relational)
  • Migrations
  • Distributed Lock (Based on atomic write operation, Standalone Package)

Transport

• InMemory (Default)
• Apache Kafka

Scalability & Fault Tolerance

• Support for multiple outbox worker instances
  • Multiple instances of the same service
  • Multiple processes in the same service
• Support for multiple inbox instances
  • Multiple instances of the same service
  • Multiple processes in the same service
• Standalone inbox
• Error handling in background services
• Dead messages
• Hook processing
  • Outbox
  • Inbox

Observability

• Support for OpenTelemetry
• Outbox size
• Inbox size
• Message failure rate
• Message delivery latency
• Number of duplicated messages
• Message duplication rate

Maintenecne

• Remove processed messages from the outbox
• Remove processed messages from the inbox
• Remove expired idempotency keys
• Archive processed messages from the outbox
• Archive processed messages from the inbox
• Correlation ID

Other

• Modular package architecture
• Support for TimeProvider
• Unordered messages
• Internal high-performance logging (Source Generators)
• Execution context in Inbox
• High-performance invoking of inbox message handlers from assemblies (Compiled Lambda Expressions)
• Grouping of messages
  • Group by topic outbox messages to a single transport message from batch (better compression)
  • Adjusting optimal transport message size
• Messages serialization and deserialization
  • System.Text.Json (default)
• Messages compression and decompression
  • No compression (default)
  • Brotli
  • GZip
• Message streaming
• Package configuration using appsetings.json
• Own transport message serialier and deseralizer (with StringBuilder)
• Idempotent messages
• Keyed in memory lock (based on SemaphoreSlim and ConcurrentDictionary)
• Transport discriminator (one outbox many transport, tagged message)
• Hook startup (checking if there are messages to be processed after startup)
• Encrypted transport messages
• Inbox based on the header, selects the appropriate algorithm to:
  • Decompression
  • Deserialization
  • Separator

🗺️ Roadmap

Name of TransactionalBox fits perfectly with the future of the project. It will be possible to use different transactional boxes.

Outbox and Inbox

No guarantee of message order between services.

Implementation under the competition.

e.g. Payment service asynchronously sends notification of payment to user.

Improvements:

• Code refactor
• More tests
• Support for more storage providers (e.g. MongoDB, Marten)
• Support for more transport providers (e.g. Iggy, RabbitMq, HTTP, gRPC)
• Performance optimization

StreamOubox and StreamInbox

Guarantee of message order in stream between services.

Idea
Ensuring the order of messages within a stream. Messages are sent by StreamOutbox and the order is respected in StreamInbox. Transport provider does not have to support message order.

e.g. (CQRS Pattern) When transfer is made in the write service, the event asynchronously refreshes the account balance in the read service.

InternalBox

Guarantee of message order in stream inside the service.

Idea
Ensuring the order of messages within a stream insite the service. Messages are added to storage provider and then processed. Transport provider is unnecessary.

e.g. Asynchronous internal communication between Aggregate Roots using domain events in the same service.

🏅 Competition 100commitow

The project is part of the competition 100 commitow.

Topics

• Distributed lock (prevent race condition)
• Hook processing
• Concurrency control
• Scaling and parallel processing (distributed processing)
• Synchronization primitives
• Idepotent messages (message deduplication)
• Retry Pattern