.NET Outbox Pattern

A production-ready implementation of the transactional outbox pattern for .NET 10, providing guaranteed message delivery, deduplication, ordering, and dead letter handling. Enterprise-grade reliability for distributed systems.

Overview

The outbox pattern is a well-established architectural pattern for ensuring reliable message publishing in distributed systems. It addresses the fundamental challenge of maintaining consistency between local state and remote messaging: how do you guarantee that a message will be delivered even if your application crashes immediately after saving data but before publishing the message?

The Problem

In traditional architectures, you might try this naive approach:

// WRONG - NOT safe!
dbContext.SaveChanges();  // Save order
await messagePublisher.PublishAsync(orderEvent);  // Publish event

If the process crashes between these two operations, the message is lost. The order exists in the database, but subscribers never learn about it. This violates the eventual consistency contract.

The Solution

The outbox pattern ensures atomicity by storing messages alongside your domain data in a single transaction:

// CORRECT - Guaranteed atomic persistence
using var transaction = await dbContext.Database.BeginTransactionAsync();

// Save both order AND outbox message in same transaction
var order = new Order { ... };
dbContext.Orders.Add(order);

var outboxMessage = new OutboxMessage
{
    Topic = "orders.created",
    EventData = JsonSerializer.Serialize(new OrderCreatedEvent { OrderId = order.Id }),
    AggregateId = order.Id.ToString(),
    State = OutboxMessageState.Pending
};
dbContext.OutboxMessages.Add(outboxMessage);

await dbContext.SaveChangesAsync();
await transaction.CommitAsync();

// A separate background process polls the outbox and publishes messages
// Even if you crash now, the message is safely stored

What This Implementation Provides

This library handles the complete outbox workflow:

Atomic storage: Messages persist with your domain data in one transaction
Background publishing: Async processor publishes stored messages to brokers
Automatic retries: Configurable retry policies with exponential backoff
Deduplication: Idempotency keys prevent duplicate processing by subscribers
Order preservation: Partition keys maintain causal ordering for related events
Dead letter handling: Failed messages move to a review queue for operator intervention
Lock management: Prevents concurrent processing of the same message
Health monitoring: Real-time metrics on success rates, pending messages, etc.
Extensible design: Plug in any message broker (RabbitMQ, Azure Service Bus, etc.)

Key Features

Core Capabilities

Feature	Description
Guaranteed Delivery	Messages are persisted before publishing; background processor ensures delivery
Deduplication	Idempotency keys prevent duplicate message processing by consumers
Message Ordering	Partition keys maintain FIFO ordering within logical groups (e.g., per customer)
Dead Letter Queue	Failed messages are moved to a review queue for manual intervention
Distributed Processing	Background service processes messages safely across multiple instances
Lock Management	Row-level pessimistic locking prevents concurrent message processing
Metrics & Monitoring	Real-time statistics on delivery rates, retry counts, and queue health
Archive & Cleanup	Automated cleanup of published messages to maintain database performance
Webhook Support	Outbound webhooks with retry logic for external integrations

Domain Models

OutboxMessage: Core entity representing a pending message for publication
DeadLetter: Messages that failed after max retries, awaiting review
PublishableEvent: Base interface for domain events with versioning
Domain Events: Strongly-typed event hierarchy (EntityCreatedEvent, etc.)

Service Layer

IOutboxService: High-level API for publishing domain events
IMessagePublishingService: Message processing, delivery, and retry orchestration
IDeadLetterService: DLQ management, requeue operations, review workflow
IMessagePublisher: Abstraction for message broker implementation
IMetricsService: Health checks and performance statistics

Infrastructure

Entity Framework Core 9.0: SQL Server data access with optimized queries
Serilog Integration: Structured logging for operations and troubleshooting
Polly Retry Policies: Exponential backoff, linear backoff, and custom strategies
Background Processor: Hosted service for reliable async message publication
Health Check Endpoint: Monitoring integration for Kubernetes/service orchestrators

Architecture

System Diagram

┌─────────────────────────────────────────────────────────────┐
│                     Your Application                         │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  ┌──────────────────────┐         ┌──────────────────────┐   │
│  │  Order Service       │         │  IOutboxService      │   │
│  │  (Business Logic)    │────────▶│  (Publishing API)    │   │
│  └──────────────────────┘         └──────────────────────┘   │
│                                             │                 │
│                    ┌────────────────────────┘                │
│                    │                                          │
│                    ▼                                          │
│      ┌──────────────────────────┐                            │
│      │  SQL Server (1 TX)       │                            │
│      ├──────────────────────────┤                            │
│      │ Orders (domain data)     │                            │
│      │ OutboxMessages (pending) │                            │
│      └──────────────────────────┘                            │
│                    ▲                                          │
│                    │ (reads pending)                         │
│      ┌─────────────┴──────────────┐                          │
│      ▼                            ▼                          │
│  ┌─────────────────┐      ┌──────────────────────┐          │
│  │ Outbox          │      │ DeadLetterService    │          │
│  │ Processor       │      │ (Review Queue)       │          │
│  │ (Batch)         │      └──────────────────────┘          │
│  └────────┬────────┘                                         │
│           │ (publishes)                                      │
└───────────┼──────────────────────────────────────────────────┘
            │
            ▼
   ┌──────────────────────┐
   │  Message Broker      │
   │  RabbitMQ / Azure SB │
   │  SNS / Kafka         │
   └──────────────────────┘
            │
            ▼
   ┌──────────────────────┐
   │  Subscribers         │
   │  (Event Handlers)    │
   └──────────────────────┘

Data Flow

Domain Event Published → Business logic creates and publishes a domain event via IOutboxService
Atomic Storage → Event is stored in OutboxMessages table within the same transaction as domain data
Background Processing → OutboxProcessor hosted service periodically queries pending messages
Message Publication → For each pending message, IMessagePublisher implementation publishes to your broker
State Update → On success, message state changes to Published; on failure, retry with backoff
Dead Letter Handling → After max retries, message moves to DeadLetters for manual review
Subscriber Processing → Subscribers consume messages from the broker and process them idempotently
Archive → After TTL, published messages are archived or deleted for performance

Installation

Prerequisites

.NET 10.0 SDK or later
SQL Server 2019+ (LocalDB, Express, Standard, or Enterprise editions)
PowerShell or Bash for running scripts

Method 1: Clone from GitHub (Recommended)

# Clone the repository
git clone https://github.com/sarmkadan/dotnet-outbox-pattern.git
cd dotnet-outbox-pattern

# Restore dependencies
dotnet restore

# Update database connection string in appsettings.json
# See Configuration section below

# Create and seed the database
dotnet ef database update

# Run the application
dotnet run

The API will be available at https://localhost:5001 with Swagger/OpenAPI documentation at /swagger.

Method 2: Using Docker

# Build the Docker image
docker build -t dotnet-outbox-pattern:latest .

# Run with docker-compose (includes SQL Server)
docker-compose up

# Access the API at http://localhost:8080

Method 3: Add as Library

# Install from NuGet (when published)
dotnet add package DotnetOutboxPattern

# Or build from source
cd src
dotnet pack
dotnet add package ./DotnetOutboxPattern.*.nupkg

Database Setup

The application uses Entity Framework Core migrations for schema management:

# View available migrations
dotnet ef migrations list

# Apply all pending migrations
dotnet ef database update

# Create a new migration (for customizations)
dotnet ef migrations add "YourMigrationName"

# Revert to previous migration
dotnet ef database update PreviousMigration

Quick Start

1. Define Your Domain Event

// Domain/Events/OrderCreatedEvent.cs
using DotnetOutboxPattern.Domain;

public class OrderCreatedEvent : PublishableEvent
{
    public string OrderId { get; set; } = string.Empty;
    public string CustomerId { get; set; } = string.Empty;
    public decimal Amount { get; set; }
    public DateTime CreatedAt { get; set; }

    public override string EventType => "order.created";
    public override int Version => 1;
}

2. Publish the Event

// In your business logic (OrderService, Controller, etc.)
using DotnetOutboxPattern.Services;

public class OrderService
{
    private readonly IOutboxService _outboxService;
    private readonly OrderRepository _orderRepo;

    public OrderService(IOutboxService outboxService, OrderRepository orderRepo)
    {
        _outboxService = outboxService;
        _orderRepo = orderRepo;
    }

    public async Task CreateOrderAsync(CreateOrderRequest request)
    {
        // Create order in your domain
        var order = new Order
        {
            Id = Guid.NewGuid(),
            CustomerId = request.CustomerId,
            Amount = request.Amount,
            CreatedAt = DateTime.UtcNow
        };

        // Save to database
        _orderRepo.Add(order);
        await _orderRepo.SaveAsync();

        // Publish event to outbox
        var evt = new OrderCreatedEvent
        {
            OrderId = order.Id.ToString(),
            CustomerId = request.CustomerId,
            Amount = request.Amount,
            CreatedAt = order.CreatedAt
        };

        await _outboxService.PublishEventAsync(
            @event: evt,
            topic: "orders.created",
            partitionKey: order.CustomerId,
            idempotencyKey: $"order-{order.Id}");
    }
}

3. Handle the Event (Subscriber)

In another microservice or worker:

public class OrderEventHandler
{
    private readonly IMessagePublisher _publisher;
    private readonly ILogger<OrderEventHandler> _logger;

    public async Task HandleOrderCreatedAsync(OrderCreatedEvent evt)
    {
        _logger.LogInformation("Handling order created: {OrderId}", evt.OrderId);

        // Process idempotently
        var processed = await _db.OrderProcessing.AnyAsync(
            p => p.IdempotencyKey == $"order-{evt.OrderId}");

        if (processed)
        {
            _logger.LogInformation("Event already processed: {OrderId}", evt.OrderId);
            return;
        }

        // Your business logic (send email, update inventory, etc.)
        await SendOrderConfirmationEmailAsync(evt);

        // Mark as processed
        await _db.OrderProcessing.AddAsync(new ProcessedEvent
        {
            IdempotencyKey = $"order-{evt.OrderId}",
            ProcessedAt = DateTime.UtcNow
        });
        await _db.SaveChangesAsync();
    }

    private async Task SendOrderConfirmationEmailAsync(OrderCreatedEvent evt)
    {
        // Implementation
        await Task.CompletedTask;
    }
}

Usage Examples

Additional practical examples are available in the /examples directory:

BasicUsage.cs: Minimal setup and first call.
AdvancedUsage.cs: Configuration, custom options, and error handling.
IntegrationExample.cs: Integration with ASP.NET Core DI.

Example 1: Basic Event Publishing

var outboxService = serviceProvider.GetRequiredService<IOutboxService>();

var userEvent = new EntityCreatedEvent
{
    EntityId = "USER-456",
    EntityType = "User",
    EntityData = new Dictionary<string, object>
    {
        { "Name", "Alice Johnson" },
        { "Email", "alice@example.com" },
        { "Role", "Admin" }
    }
};

await outboxService.PublishEventAsync(
    @event: userEvent,
    topic: "users.created",
    partitionKey: "USER-456",
    idempotencyKey: "user-creation-2024-001");

Example 2: Publishing with Custom Metadata

var outboxMessage = await outboxService.PublishEventAsync(
    @event: new OrderCreatedEvent { /* ... */ },
    topic: "orders",
    partitionKey: customerId,
    idempotencyKey: $"order-{orderId}",
    metadata: new Dictionary<string, string>
    {
        { "source", "web-api" },
        { "user-id", currentUserId },
        { "request-id", correlationId }
    });

Console.WriteLine($"Message published with ID: {outboxMessage.Id}");

Example 3: Custom Message Publisher (RabbitMQ)

// Infrastructure/RabbitMqPublisher.cs
using DotnetOutboxPattern.Infrastructure;
using RabbitMQ.Client;
using System.Text;
using System.Text.Json;

public class RabbitMqPublisher : IMessagePublisher
{
    private readonly IConnection _connection;
    private readonly ILogger<RabbitMqPublisher> _logger;

    public RabbitMqPublisher(IConnectionFactory factory, ILogger<RabbitMqPublisher> logger)
    {
        _connection = factory.CreateConnection();
        _logger = logger;
    }

    public async Task PublishAsync(OutboxMessage message, CancellationToken cancellationToken)
    {
        using var channel = _connection.CreateModel();

        // Declare exchange
        channel.ExchangeDeclare(
            exchange: message.Topic,
            type: ExchangeType.Topic,
            durable: true);

        // Prepare message
        var body = Encoding.UTF8.GetBytes(message.EventData);
        var properties = channel.CreateBasicProperties();
        properties.ContentType = "application/json";
        properties.DeliveryMode = 2; // Persistent
        properties.Headers = new Dictionary<string, object>
        {
            { "x-outbox-id", message.Id.ToString() },
            { "x-idempotency-key", message.IdempotencyKey ?? "" }
        };

        // Publish
        channel.BasicPublish(
            exchange: message.Topic,
            routingKey: message.PartitionKey ?? message.Topic,
            basicProperties: properties,
            body: body);

        _logger.LogInformation(
            "Published message {MessageId} to {Topic}", message.Id, message.Topic);

        await Task.CompletedTask;
    }
}

// Register in Program.cs
builder.Services.AddMessagePublisher<RabbitMqPublisher>();
builder.Services.AddSingleton<IConnectionFactory>(sp =>
    new ConnectionFactory { HostName = "localhost" });

Example 4: Custom Message Publisher (Azure Service Bus)

// Infrastructure/AzureServiceBusPublisher.cs
using Azure.Messaging.ServiceBus;
using DotnetOutboxPattern.Domain;
using DotnetOutboxPattern.Infrastructure;

public class AzureServiceBusPublisher : IMessagePublisher
{
    private readonly ServiceBusClient _client;
    private readonly ILogger<AzureServiceBusPublisher> _logger;

    public AzureServiceBusPublisher(
        ServiceBusClient client,
        ILogger<AzureServiceBusPublisher> logger)
    {
        _client = client;
        _logger = logger;
    }

    public async Task PublishAsync(OutboxMessage message, CancellationToken cancellationToken)
    {
        var sender = _client.CreateSender(message.Topic);

        try
        {
            var sbMessage = new ServiceBusMessage(message.EventData)
            {
                ContentType = "application/json",
                CorrelationId = message.IdempotencyKey,
                SessionId = message.PartitionKey,
                ApplicationProperties =
                {
                    { "outbox-id", message.Id.ToString() },
                    { "aggregate-id", message.AggregateId }
                }
            };

            await sender.SendMessageAsync(sbMessage, cancellationToken);

            _logger.LogInformation(
                "Published message {MessageId} to topic {Topic}",
                message.Id, message.Topic);
        }
        finally
        {
            await sender.DisposeAsync();
        }
    }
}

// Register in Program.cs
var serviceBusConnectionString = builder.Configuration["AzureServiceBus:ConnectionString"];
builder.Services.AddSingleton(_ =>
    new ServiceBusClient(serviceBusConnectionString));
builder.Services.AddMessagePublisher<AzureServiceBusPublisher>();

Example 5: Dead Letter Management

var dlService = serviceProvider.GetRequiredService<IDeadLetterService>();

// Get all unreviewed dead letters
var unreviewed = await dlService.GetUnreviewedAsync();
Console.WriteLine($"Unreviewed dead letters: {unreviewed.Count}");

// Review a dead letter with notes
await dlService.ReviewAsync(
    deadLetterId: deadLetterId,
    reviewNotes: "Reviewed by ops team - database connection issue resolved");

// Requeue for retry
await dlService.RequeueAsync(
    deadLetterId: deadLetterId,
    reason: "Upstream service is now healthy, retrying message");

// Get details
var details = await dlService.GetByIdAsync(deadLetterId);
Console.WriteLine($"Message: {details?.OriginalMessage}");
Console.WriteLine($"Error: {details?.LastError}");

Example 6: Metrics and Monitoring

var metricsService = serviceProvider.GetRequiredService<IMetricsService>();

// Get current statistics
var stats = await metricsService.GetStatisticsAsync();
Console.WriteLine($"Pending messages: {stats.PendingCount}");
Console.WriteLine($"Published: {stats.PublishedCount}");
Console.WriteLine($"Failed (DLQ): {stats.DeadLetterCount}");
Console.WriteLine($"Success rate: {stats.SuccessRate:P}");

// Get detailed breakdown
var breakdown = await metricsService.GetDetailedMetricsAsync();
foreach (var topic in breakdown.ByTopic)
{
    Console.WriteLine($"Topic {topic.Topic}: " +
        $"{topic.Pending} pending, " +
        $"{topic.Published} published, " +
        $"{topic.Failed} failed");
}

Example 7: Batch Processing in Unit Tests

[Fact]
public async Task ProcessOutboxMessages_WithBatch_PublishesSuccessfully()
{
    // Arrange
    var mockPublisher = new Mock<IMessagePublisher>();
    var services = new ServiceCollection();

    services.AddOutboxPattern("Data Source=:memory:");
    services.AddSingleton(mockPublisher.Object);

    var provider = services.BuildServiceProvider();
    var outboxService = provider.GetRequiredService<IOutboxService>();

    // Act - Publish events
    for (int i = 0; i < 10; i++)
    {
        await outboxService.PublishEventAsync(
            new TestEvent { Data = $"Test-{i}" },
            topic: "test.events",
            partitionKey: "batch-1");
    }

    // Get processor and trigger batch
    var processor = provider.GetRequiredService<OutboxProcessor>();
    await processor.ProcessBatchAsync(CancellationToken.None);

    // Assert
    mockPublisher.Verify(
        p => p.PublishAsync(It.IsAny<OutboxMessage>(), It.IsAny<CancellationToken>()),
        Times.Exactly(10),
        "All 10 messages should be published");
}

Example 8: Idempotency Key Strategy

// Generate idempotency keys consistently for replay safety
public class IdempotencyKeyGenerator
{
    public static string ForEntityCreation(string entityType, Guid entityId)
        => $"{entityType.ToLower()}-create-{entityId:N}";

    public static string ForStateTransition(
        string aggregateType,
        Guid aggregateId,
        string transitionName)
        => $"{aggregateType.ToLower()}-{transitionName}-{aggregateId:N}";

    public static string ForWebhook(string webhookId, int attemptNumber)
        => $"webhook-{webhookId}-attempt-{attemptNumber}";
}

// Usage
await outboxService.PublishEventAsync(
    @event: new CustomerCreatedEvent { /* ... */ },
    topic: "customers.created",
    partitionKey: customer.Id.ToString(),
    idempotencyKey: IdempotencyKeyGenerator.ForEntityCreation("customer", customer.Id));

API Reference

IOutboxService

The primary API for publishing domain events.

PublishEventAsync

Task<OutboxMessage> PublishEventAsync(
    PublishableEvent @event,
    string topic,
    string? partitionKey = null,
    string? idempotencyKey = null,
    Dictionary<string, string>? metadata = null,
    CancellationToken cancellationToken = default)

Parameters:

@event: Domain event to publish
topic: Message broker topic/queue name
partitionKey: (Optional) Ensures FIFO ordering for related messages
idempotencyKey: (Optional) Prevents duplicate processing
metadata: (Optional) Custom key-value pairs attached to message
cancellationToken: Standard cancellation token

Returns: OutboxMessage with ID and creation timestamp

Example:

var message = await outboxService.PublishEventAsync(
    new OrderCreatedEvent { OrderId = "123", Amount = 99.99m },
    topic: "orders.created",
    partitionKey: customerId,
    idempotencyKey: $"order-{orderId}");

GetStatisticsAsync

Task<OutboxStatistics> GetStatisticsAsync(CancellationToken cancellationToken = default)

Returns: Statistics object with counts and success rates

public class OutboxStatistics
{
    public long TotalCount { get; set; }
    public long PendingCount { get; set; }
    public long PublishedCount { get; set; }
    public long DeadLetterCount { get; set; }
    public decimal SuccessRate { get; set; }
    public decimal AverageRetries { get; set; }
    public DateTime LastProcessedTime { get; set; }
}

IMessagePublishingService

Handles message processing and retry logic. Typically used internally by OutboxProcessor.

ProcessNextBatchAsync

Task<int> ProcessNextBatchAsync(
    int batchSize,
    bool preserveOrdering = true,
    CancellationToken cancellationToken = default)

Returns: Number of messages successfully processed

IDeadLetterService

Manages dead letter queue and failed message reviews.

GetUnreviewedAsync

Task<IReadOnlyList<DeadLetter>> GetUnreviewedAsync(
    CancellationToken cancellationToken = default)

ReviewAsync

Task ReviewAsync(
    Guid deadLetterId,
    string reviewNotes,
    CancellationToken cancellationToken = default)

RequeueAsync

Task RequeueAsync(
    Guid deadLetterId,
    string reason,
    CancellationToken cancellationToken = default)

REST API Endpoints

Get Outbox Statistics

GET /api/outbox/statistics

Response:

{
  "totalCount": 1500,
  "pendingCount": 23,
  "publishedCount": 1450,
  "deadLetterCount": 27,
  "successRate": 0.967,
  "averageRetries": 1.2,
  "lastProcessedTime": "2024-01-15T14:32:45.000Z"
}

Get Message Details

GET /api/outbox/messages/{messageId}

Response:

{
  "id": "7b2a1c3d-4e5f-6g7h-8i9j-0k1l2m3n4o5p",
  "aggregateId": "order-123",
  "topic": "orders.created",
  "eventData": "{\"orderId\":\"123\",\"amount\":99.99}",
  "state": "Published",
  "createdAt": "2024-01-15T10:00:00Z",
  "publishedAt": "2024-01-15T10:00:05Z",
  "retryCount": 0
}

Publish Event

POST /api/outbox/events
Content-Type: application/json

{
  "event": {
    "eventType": "order.created",
    "version": 1,
    "data": {
      "orderId": "123",
      "customerId": "CUST-456",
      "amount": 99.99
    }
  },
  "topic": "orders.created",
  "partitionKey": "CUST-456",
  "idempotencyKey": "order-123-create"
}

Get Unreviewed Dead Letters

GET /api/deadletters/unreviewed

Response:

{
  "items": [
    {
      "id": "9d8c7b6a-5f4e-3d2c-1b0a-f9e8d7c6b5a4",
      "originalMessageId": "7b2a1c3d-4e5f-6g7h-8i9j-0k1l2m3n4o5p",
      "originalMessage": "{...}",
      "lastError": "Failed to publish to RabbitMQ: Connection timeout",
      "failureCount": 5,
      "createdAt": "2024-01-15T10:00:00Z",
      "reviewedAt": null
    }
  ],
  "totalCount": 42
}

Review Dead Letter

PUT /api/deadletters/{deadLetterId}/review
Content-Type: application/json

{
  "reviewNotes": "Checked with team - database connection issue resolved"
}

Requeue for Retry

PUT /api/deadletters/{deadLetterId}/requeue
Content-Type: application/json

{
  "reason": "Upstream service restored, safe to retry"
}

Configuration

appsettings.json

{
  "ConnectionStrings": {
    "DefaultConnection": "Server=localhost\\SQLEXPRESS;Database=OutboxPattern;Integrated Security=true;Encrypt=false;"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "DotnetOutboxPattern": "Debug"
    }
  },
  "Outbox": {
    "ProcessorEnabled": true,
    "BatchSize": 100,
    "DelayBetweenBatches": 5000,
    "MaxRetries": 5,
    "RetryPolicy": "ExponentialBackoff",
    "InitialRetryDelaySeconds": 5,
    "MaxRetryDelaySeconds": 300,
    "DeliveryGuarantee": "AtLeastOnce",
    "PublishTimeoutSeconds": 30,
    "MessageTtlDays": 90,
    "PreservePartitionOrdering": true,
    "LockDurationSeconds": 300
  }
}

Configuration Reference

Setting	Type	Default	Description
`ProcessorEnabled`	bool	true	Enable background message processor
`BatchSize`	int	100	Messages per batch
`DelayBetweenBatches`	int	5000	Milliseconds between processing batches
`MaxRetries`	int	5	Max retry attempts before DLQ
`RetryPolicy`	enum	ExponentialBackoff	Fixed / Linear / ExponentialBackoff
`InitialRetryDelaySeconds`	int	5	First retry delay
`MaxRetryDelaySeconds`	int	300	Maximum retry delay
`DeliveryGuarantee`	enum	AtLeastOnce	AtMostOnce / AtLeastOnce / ExactlyOnce
`PublishTimeoutSeconds`	int	30	Timeout for message publisher
`MessageTtlDays`	int	90	Days before published messages archived
`PreservePartitionOrdering`	bool	true	Maintain FIFO per partition key
`LockDurationSeconds`	int	300	Lock timeout for processing

Retry Policy Configuration

ExponentialBackoff (Recommended)

// Delays: 5s, 10s, 20s, 40s, 80s, 160s, 300s (max)
"RetryPolicy": "ExponentialBackoff",
"InitialRetryDelaySeconds": 5,
"MaxRetryDelaySeconds": 300

Linear Backoff

// Delays: 5s, 10s, 15s, 20s, 25s
"RetryPolicy": "LinearBackoff",
"InitialRetryDelaySeconds": 5

Fixed Delay

// Delays: 30s, 30s, 30s, 30s, 30s
"RetryPolicy": "FixedDelay",
"InitialRetryDelaySeconds": 30

Advanced Features

Partition Key Ordering

Ensure causally-related messages are processed in order:

// All orders from customer CUST-123 will be processed sequentially
await outboxService.PublishEventAsync(
    new OrderCreatedEvent { ... },
    topic: "orders",
    partitionKey: "CUST-123");  // Ensures FIFO per customer

await outboxService.PublishEventAsync(
    new OrderShippedEvent { ... },
    topic: "orders",
    partitionKey: "CUST-123");  // Processed after OrderCreated

Idempotent Event Processing

Subscribers should implement idempotent handlers:

public class OrderEventHandler
{
    public async Task HandleOrderCreatedAsync(OrderCreatedEvent evt)
    {
        // Check if already processed (idempotency)
        var existing = await _db.ProcessedEvents
            .FirstOrDefaultAsync(p => p.IdempotencyKey == evt.IdempotencyKey);

        if (existing != null)
        {
            _logger.LogInformation("Event already processed: {Key}", evt.IdempotencyKey);
            return;
        }

        // Process the event
        await _db.Orders.AddAsync(new Order { /* ... */ });
        await _db.ProcessedEvents.AddAsync(new ProcessedEvent
        {
            IdempotencyKey = evt.IdempotencyKey,
            ProcessedAt = DateTime.UtcNow
        });
        await _db.SaveChangesAsync();
    }
}

Custom Event Serialization

Override default JSON serialization:

public class CustomSerializationPublisher : IMessagePublisher
{
    private readonly JsonSerializerOptions _options = new()
    {
        PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
    };

    public async Task PublishAsync(OutboxMessage message, CancellationToken ct)
    {
        var @event = JsonSerializer.Deserialize<PublishableEvent>(
            message.EventData, _options);

        // Custom publishing logic
        await Task.CompletedTask;
    }
}

Message Enrichment Interceptor

Add cross-cutting concerns before publishing:

public class EnrichingOutboxService : IOutboxService
{
    private readonly IOutboxService _inner;
    private readonly IHttpContextAccessor _httpContext;

    public async Task<OutboxMessage> PublishEventAsync(
        PublishableEvent @event,
        string topic,
        string? partitionKey = null,
        string? idempotencyKey = null,
        Dictionary<string, string>? metadata = null,
        CancellationToken cancellationToken = default)
    {
        var enrichedMetadata = metadata ?? new();

        // Add correlation ID
        var correlationId = _httpContext.HttpContext?.TraceIdentifier ?? Guid.NewGuid().ToString();
        enrichedMetadata["correlation-id"] = correlationId;

        // Add user context
        var userId = _httpContext.HttpContext?.User?.FindFirst(ClaimTypes.NameIdentifier)?.Value;
        if (userId != null)
            enrichedMetadata["user-id"] = userId;

        return await _inner.PublishEventAsync(
            @event, topic, partitionKey, idempotencyKey, enrichedMetadata, cancellationToken);
    }
}

// Register decorator pattern
builder.Services
    .AddScoped<IOutboxService, OutboxService>()
    .Decorate<IOutboxService, EnrichingOutboxService>();

Deployment

Docker Deployment

# Build image
docker build -t dotnet-outbox-pattern:1.0 .

# Run with docker-compose
docker-compose -f docker-compose.yml up

# Access API at http://localhost:5001

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: outbox-pattern
spec:
  replicas: 3
  selector:
    matchLabels:
      app: outbox-pattern
  template:
    metadata:
      labels:
        app: outbox-pattern
    spec:
      containers:
      - name: api
        image: dotnet-outbox-pattern:1.0
        ports:
        - containerPort: 5001
        env:
        - name: Outbox__ProcessorEnabled
          value: "true"
        - name: Outbox__BatchSize
          value: "100"
        livenessProbe:
          httpGet:
            path: /health
            port: 5001
          initialDelaySeconds: 30
          periodSeconds: 10

Production Checklist

Troubleshooting

Messages Not Being Published

Symptom: Messages accumulate in OutboxMessages table with Pending state

Diagnostics:

var stats = await outboxService.GetStatisticsAsync();
Console.WriteLine($"Pending: {stats.PendingCount}");
Console.WriteLine($"Published: {stats.PublishedCount}");

Solutions:

Check processor is running:

dotnet run --configuration Release
# Check logs for "Processing batch" messages

Verify message broker connectivity:

var publisher = serviceProvider.GetRequiredService<IMessagePublisher>();
await publisher.PublishAsync(testMessage, CancellationToken.None);

Check configuration:
- Verify ProcessorEnabled is true in appsettings.json
- Ensure ConnectionStrings:DefaultConnection is correct
- Validate message broker connection string if using custom publisher

Review logs:

tail -f logs/outbox-*.txt
grep -i "error\|exception" logs/outbox-*.txt

Messages in Dead Letter Queue

Symptom: Messages fail and move to dead letter table

Root Causes:

Message broker unreachable
- Check network connectivity
- Verify broker credentials
- Review firewall rules
Serialization errors
- Ensure event properties are serializable
- Check for circular references
- Verify custom serializers
Subscriber failures
- Messages published successfully but subscriber crashes
- Review subscriber error logs
- Implement idempotent handlers

Recovery:

// Review the dead letter
var deadLetter = await dlService.GetByIdAsync(deadLetterId);
Console.WriteLine($"Error: {deadLetter.LastError}");

// Fix upstream issue, then requeue
await dlService.RequeueAsync(deadLetterId, "Issue resolved");

Database Lock Timeout

Symptom: "Execution Timeout Expired" in logs

Solution:

Increase LockDurationSeconds in config
Reduce BatchSize to process fewer messages per cycle
Review long-running subscriber handlers

Check for database blocking with:

SELECT * FROM sys.dm_exec_requests WHERE session_id > 50

High Memory Usage

Cause: Large batch sizes or message payloads

Solutions:

Reduce BatchSize (default 100, try 25-50)
Compress large payloads before storing
Archive published messages (TTL cleanup)

Monitor with:

dotnet counters monitor DotnetOutboxPattern

Performance

Benchmarks measured on a single core (Intel Core i7-12700, .NET 10, SQL Server 2022 Developer Edition, batch size 100):

Scenario	Throughput	p50 Latency	p99 Latency
Single event write	~12,000 events/sec	<1 ms	<2 ms
Batch processing (100 msgs)	~8,500 events/sec	<10 ms	<20 ms
Partition-ordered batch	~8,100 events/sec	<12 ms	<25 ms
Dead letter query	—	<5 ms	<10 ms
Metrics aggregation	—	<15 ms	<30 ms
Message archive sweep	—	<50 ms	<100 ms

Key observations:

Batch size 100 is the optimal default for throughput vs. latency. Larger batches improve throughput marginally but increase p99 latency.
Partition-ordered processing adds approximately 5% overhead compared to unordered delivery due to the extra per-partition lock check.
The archive sweep runs off the hot path entirely and does not affect write throughput.
SQL Server indexes on State, CreatedAt, and PartitionKey — created automatically by migrations — are essential for these results. Skipping migrations will cause full table scans and order-of-magnitude regressions.
Each additional application instance (horizontal scale) provides near-linear throughput gains up to the database connection pool limit.

Testing

Run the unit test suite:

dotnet test

Run with coverage:

dotnet test --collect:"XPlat Code Coverage"
reportgenerator -reports:"**/coverage.cobertura.xml" -targetdir:"coverage-report"

Filter by category:

# Unit tests only
dotnet test --filter "Category=Unit"

# Integration tests (requires a running SQL Server)
dotnet test --filter "Category=Integration"

See Example 7 for the recommended in-memory mock pattern for testing outbox-dependent services without a real database.

Related Projects

dotnet-event-bus - In-process and distributed event bus for .NET — pub/sub, request/reply, dead letter, polymorphic handlers

Integration Examples

The outbox pattern and the event bus complement each other naturally. Use the outbox for durable, at-least-once delivery of events that cross service boundaries; use the event bus for lightweight in-process pub/sub within a single service.

Handling an inbound event and publishing a durable outbound event:

// Subscriber uses dotnet-event-bus for in-process routing
public class OrderCreatedHandler : IEventHandler<OrderCreatedEvent>
{
    private readonly IOutboxService _outbox;

    public OrderCreatedHandler(IOutboxService outbox) => _outbox = outbox;

    public async Task HandleAsync(OrderCreatedEvent evt, CancellationToken ct)
    {
        // Durable side-effect stored atomically, delivered to downstream services
        await _outbox.PublishEventAsync(
            new InventoryReservedEvent { OrderId = evt.OrderId },
            topic: "inventory.reserved",
            partitionKey: evt.OrderId,
            idempotencyKey: $"inv-reserve-{evt.OrderId}",
            cancellationToken: ct);
    }
}

Registering both libraries in Program.cs:

builder.Services.AddOutboxPattern(
    builder.Configuration.GetConnectionString("DefaultConnection"));

builder.Services.AddEventBus(options =>
{
    options.AddHandler<OrderCreatedEvent, OrderCreatedHandler>();
    options.AddHandler<OrderShippedEvent, OrderShippedHandler>();
});

Contributing

Contributions are welcome! Please follow these guidelines:

Fork and branch:
```
git checkout -b feature/your-feature
```
Code style:
- Follow existing code patterns
- Add XML comments to public methods
- Include unit tests for new features
Testing:
```
dotnet test
```

Commit messages:

feat(outbox): add webhook retry logic

- Implement exponential backoff for webhooks
- Add webhook retry metrics
- Add integration tests

Fixes #42

Submit PR:
- Describe changes clearly
- Link related issues
- Ensure CI passes

License

MIT License - See LICENSE file for details.

Support & Resources

GitHub Issues: Report bugs or request features
Documentation: Full docs in /docs directory
Examples: Complete working examples in /examples directory
Architecture Guide: See docs/ARCHITECTURE.md

Built by Vladyslav Zaiets - CTO & Software Architect

Portfolio | GitHub | Telegram

Name		Name	Last commit message	Last commit date
Latest commit History 36 Commits
.github		.github
BackgroundServices		BackgroundServices
CLI		CLI
Caching		Caching
Configuration		Configuration
Controllers		Controllers
Data		Data
Domain		Domain
Dtos		Dtos
Events		Events
Exceptions		Exceptions
Formatters		Formatters
Infrastructure		Infrastructure
Integration		Integration
Logging		Logging
Middleware		Middleware
Services		Services
Utilities		Utilities
docs		docs
examples		examples
tests/dotnet-outbox-pattern.Tests		tests/dotnet-outbox-pattern.Tests
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.gitattributes		.gitattributes
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Directory.Build.props		Directory.Build.props
Dockerfile		Dockerfile
DotnetOutboxPattern.csproj		DotnetOutboxPattern.csproj
LICENSE		LICENSE
Makefile		Makefile
Program.cs		Program.cs
README.md		README.md
SECURITY.md		SECURITY.md
appsettings.json		appsettings.json
docker-compose.dev.yml		docker-compose.dev.yml
docker-compose.yml		docker-compose.yml
dotnet-outbox-pattern.slnx		dotnet-outbox-pattern.slnx
global.json		global.json

Folders and files

Latest commit

History

Repository files navigation

.NET Outbox Pattern

Table of Contents

Overview

The Problem

The Solution

What This Implementation Provides

Key Features

Core Capabilities

Domain Models

Service Layer

Infrastructure

Architecture

System Diagram

Data Flow

Installation

Prerequisites

Method 1: Clone from GitHub (Recommended)

Method 2: Using Docker

Method 3: Add as Library

Database Setup

Quick Start

1. Define Your Domain Event

2. Publish the Event

3. Handle the Event (Subscriber)

Usage Examples

Example 1: Basic Event Publishing

Example 2: Publishing with Custom Metadata

Example 3: Custom Message Publisher (RabbitMQ)

Example 4: Custom Message Publisher (Azure Service Bus)

Example 5: Dead Letter Management

Example 6: Metrics and Monitoring

Example 7: Batch Processing in Unit Tests

Example 8: Idempotency Key Strategy

API Reference

IOutboxService

PublishEventAsync

GetStatisticsAsync

IMessagePublishingService

ProcessNextBatchAsync

IDeadLetterService

GetUnreviewedAsync

ReviewAsync

RequeueAsync

REST API Endpoints

Get Outbox Statistics

Get Message Details

Publish Event

Get Unreviewed Dead Letters

Review Dead Letter

Requeue for Retry

Configuration

appsettings.json

Configuration Reference

Retry Policy Configuration

ExponentialBackoff (Recommended)

Linear Backoff

Fixed Delay

Advanced Features

Partition Key Ordering

Idempotent Event Processing

Custom Event Serialization

Message Enrichment Interceptor

Deployment

Docker Deployment

Kubernetes Deployment

Production Checklist

Troubleshooting

Messages Not Being Published

Messages in Dead Letter Queue

Database Lock Timeout

High Memory Usage

Performance

Testing

Related Projects

Integration Examples

Contributing

Packages