feat: Implement Polly Retry Policies for External Services (Issue #107)

[artcava]

📋 Overview

Implements Phase 8.1: Polly Retry Policies for handling transient failures in external services (HTTP clients, database operations, message broker). This provides a comprehensive infrastructure-level retry mechanism that complements the existing process-level retry logic.

🎯 Objectives Completed

• ✅ Install and configure Polly library
• ✅ Implement retry policies for HTTP clients
• ✅ Implement retry policies for database operations
• ✅ Implement retry policies for message broker
• ✅ Configure exponential backoff with jitter
• ✅ Define retry limits per service type
• ✅ Integrate policies with DI container
• ✅ Add comprehensive logging for retry attempts
• ✅ Write unit tests for retry behavior
• ✅ Document retry policy configuration

📦 Deliverables

1. Core Infrastructure Components

RetryPolicyConfiguration (src/StarGate.Infrastructure/Resilience/RetryPolicyConfiguration.cs)

• Configurable retry behavior parameters
• Exponential backoff calculation with jitter (±10%)
• Default values: 3 retries, 1s initial delay, 30s max delay
• CalculateDelay(int retryAttempt) method for delay computation

RetryPolicyFactory (src/StarGate.Infrastructure/Resilience/RetryPolicyFactory.cs)

• CreateHttpRetryPolicy: Handles HttpRequestException, TimeoutException, non-success status codes
• CreateDatabaseRetryPolicy: Handles TimeoutException, IOException, connection errors
• CreateBrokerRetryPolicy: Handles TimeoutException, IOException, connection errors
• CreateGenericRetryPolicy: Handles any transient exception
• Comprehensive logging for all retry attempts

ResilienceServiceCollectionExtensions (src/StarGate.Infrastructure/Extensions/ResilienceServiceCollectionExtensions.cs)

• AddResiliencePolicies: Registers retry policies in DI container
• AddHttpClientWithRetry<TClient>: Configures HTTP client with automatic retry
• Singleton registration for database and broker policies

2. Configuration Files

Production Configuration (src/StarGate.Server/appsettings.json)

"Resilience": {
  "Retry": {
    "MaxRetryAttempts": 3,
    "InitialDelaySeconds": 1.0,
    "MaxDelaySeconds": 30.0,
    "BackoffMultiplier": 2.0,
    "UseJitter": true
  }
}

Development Configuration (src/StarGate.Server/appsettings.Development.json)

"Resilience": {
  "Retry": {
    "MaxRetryAttempts": 2,
    "InitialDelaySeconds": 0.5,
    "MaxDelaySeconds": 10.0,
    "BackoffMultiplier": 2.0,
    "UseJitter": true
  }
}

Rationale: Development uses fewer retries and shorter delays for faster feedback during development.

3. NuGet Packages

Added to src/StarGate.Infrastructure/StarGate.Infrastructure.csproj:

• Polly (v8.4.2): Core retry policy library
• Polly.Extensions.Http (v3.0.0): HTTP client integration

4. Program.cs Integration

Updated src/StarGate.Server/Program.cs:

// Add resilience policies
builder.Services.AddResiliencePolicies(builder.Configuration);

5. Unit Tests

RetryPolicyConfigurationTests (tests/StarGate.Infrastructure.Tests/Resilience/RetryPolicyConfigurationTests.cs)

• ✅ Exponential backoff calculation
• ✅ Max delay enforcement
• ✅ Jitter randomization (±10%)
• ✅ Non-negative delay guarantee
• ✅ Default value verification
• ✅ Custom configuration scenarios

RetryPolicyFactoryTests (tests/StarGate.Infrastructure.Tests/Resilience/RetryPolicyFactoryTests.cs)

• ✅ HTTP policy retries on HttpRequestException and TimeoutException
• ✅ Database policy retries on TimeoutException, IOException, connection errors
• ✅ Broker policy retries on transient failures
• ✅ Generic policy handles transient exceptions
• ✅ Eventual success after failures
• ✅ Non-transient exceptions not retried
• ✅ Respect for MaxRetryAttempts configuration

Test Coverage: 13 comprehensive unit tests covering all scenarios

6. Documentation

POLLY-RETRY-POLICIES.md (docs/POLLY-RETRY-POLICIES.md)

Comprehensive 600+ line documentation covering:

• Architecture and component overview
• Two-level retry strategy (Infrastructure vs Application)
• Exponential backoff formula with examples
• Jitter implementation and rationale
• Error classification (transient vs permanent)
• Configuration examples and recommendations
• Usage examples for MongoDB and RabbitMQ
• Testing instructions (unit and integration)
• Monitoring and observability guidelines
• Performance considerations
• Troubleshooting guide
• Future enhancements roadmap

🏗️ Architecture

Two-Level Retry Strategy

This implementation creates a two-level retry system:

Level 1: Infrastructure Retry (Polly) - This PR

• Fast retries (1s → 2s → 4s = 7s total)
• Handles transient failures in MongoDB, RabbitMQ, HTTP
• Transparent to business logic
• Located in StarGate.Infrastructure.Resilience

Level 2: Application Retry (ProcessWorker) - Existing

• Slower retries (5s → 10s → 20s = 35s+ total)
• Handles complete process execution failures
• Changes process status to "Retrying"
• Located in StarGate.Server.Workers
• Documented in docs/RETRY-LOGIC.md

Exponential Backoff Formula

Delay = InitialDelay × (Multiplier ^ (RetryAttempt - 1))
Delay = min(Delay, MaxDelay)

With Jitter:
Jitter = Delay × 0.2 × (Random - 0.5)  // ±10%
FinalDelay = Delay + Jitter

Example (InitialDelay=1s, Multiplier=2.0):

• Retry 1: 1.0s ± 0.1s = 0.9s - 1.1s
• Retry 2: 2.0s ± 0.2s = 1.8s - 2.2s
• Retry 3: 4.0s ± 0.4s = 3.6s - 4.4s

Total Time: ~7 seconds for 3 retries

Why Jitter?

Without Jitter: All failed requests retry at the same time → thundering herd problem

With Jitter: Retries distributed over time → smooth load distribution → better recovery

✅ Acceptance Criteria (from Issue #107)

• ✅ Polly packages installed and configured
• ✅ RetryPolicyConfiguration implemented with exponential backoff and jitter
• ✅ RetryPolicyFactory created with policies for HTTP, database, and broker
• ✅ Generic retry policy for transient exceptions
• ✅ Retry policies registered in DI container
• ✅ Configuration files updated with retry settings
• ✅ Different settings for Development and Production
• ✅ Comprehensive logging for retry attempts
• ✅ Unit tests for RetryPolicyConfiguration
• ✅ Unit tests for RetryPolicyFactory
• ✅ Code follows CODING-CONVENTIONS.md
• ✅ Comprehensive documentation (POLLY-RETRY-POLICIES.md)

📝 Testing Instructions

Run Unit Tests

# Run all retry policy tests
dotnet test tests/StarGate.Infrastructure.Tests --filter "FullyQualifiedName~Resilience"

# Run specific test classes
dotnet test tests/StarGate.Infrastructure.Tests --filter "FullyQualifiedName~RetryPolicyConfigurationTests"
dotnet test tests/StarGate.Infrastructure.Tests --filter "FullyQualifiedName~RetryPolicyFactoryTests"

Integration Testing

Test MongoDB Retry

# 1. Start MongoDB
docker-compose up -d mongodb

# 2. Stop MongoDB to simulate failure
docker-compose stop mongodb

# 3. Try to create process (should retry and eventually fail)
POST /api/processes

# 4. Check logs for retry attempts:
# "Database retry attempt 1/3: Exception=TimeoutException, Delay=1000ms"
# "Database retry attempt 2/3: Exception=TimeoutException, Delay=2000ms"
# "Database retry attempt 3/3: Exception=TimeoutException, Delay=4000ms"

# 5. Restart MongoDB
docker-compose start mongodb

# 6. Verify requests succeed

Test RabbitMQ Retry

# 1. Stop RabbitMQ during process creation
docker-compose stop rabbitmq

# 2. Create process (should retry broker operations)
POST /api/processes

# 3. Check logs for broker retry attempts
docker logs stargate-server | grep "Broker retry attempt"

Test Exponential Backoff

• Verify in logs that delays increase: 1s → 2s → 4s
• Confirm jitter causes slight variance (not exactly 1s, 2s, 4s)

📊 Performance Impact

Success Case

• Overhead: <1ms (policy check is fast)
• No impact on successful operations

Failure Case

• Additional Latency: Up to 7 seconds (1s + 2s + 4s)
• Trade-off: Better than complete failure
• Acceptable for transient failures

🔗 Related Issues

• Closes Phase 8.1: Implement Polly Retry Policies for External Services #107
• Prerequisite for Phase 8.1: Implement Polly Circuit Breaker Pattern #108 (Circuit Breaker Integration)
• Complements existing ProcessWorker retry (Phase 7.1: Integrate Retry Logic in ProcessWorker #102)

📌 Important Notes

Difference from Existing Retry Logic

This implementation is distinct from StarGate.Core.Configuration.RetryConfiguration:

Aspect	Polly Retry (This PR)	ProcessWorker Retry (Existing)
Layer	Infrastructure	Application
Purpose	Single operation retry	Complete process retry
Delay	1s → 2s → 4s	5s → 10s → 20s
Total Time	~7 seconds	~35 seconds
Transparency	Transparent (logs only)	Visible (status change)
Use Case	MongoDB timeout	Handler timeout

Both configurations coexist and serve complementary purposes:

1. Polly handles fast, transient infrastructure failures
2. ProcessWorker handles slower, process-level failures

Configuration Namespaces

{
  "Retry": {                    // Existing - ProcessWorker retry
    "BaseDelaySeconds": 5,
    "MaxDelaySeconds": 300
  },
  "Resilience": {               // New - Polly retry
    "Retry": {
      "MaxRetryAttempts": 3,
      "InitialDelaySeconds": 1.0,
      "MaxDelaySeconds": 30.0
    }
  }
}

Next Steps for Full Integration

While this PR provides the complete Polly infrastructure, applying retry policies to existing repositories (MongoProcessRepository, RabbitMqBroker) will be done in a follow-up PR to:

• Keep changes focused and reviewable
• Avoid conflicts with ongoing development
• Allow thorough testing of the infrastructure first

The follow-up PR will:

1. Inject AsyncRetryPolicy into MongoProcessRepository constructor
2. Wrap all MongoDB operations with _retryPolicy.ExecuteAsync()
3. Apply same pattern to RabbitMqBroker
4. Add integration tests with infrastructure failures

🧪 Test Results

All unit tests pass:

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Passed!  - Failed:     0, Passed:    13, Skipped:     0, Total:    13

📋 Checklist

• ✅ Code follows project coding conventions
• ✅ Clean Architecture principles applied
• ✅ Comprehensive error handling
• ✅ Detailed logging implemented
• ✅ Configuration examples provided
• ✅ Unit tests included and passing
• ✅ Documentation complete and comprehensive
• ✅ Commit messages follow conventional commits
• ✅ No compilation errors
• ✅ No breaking changes

---

Estimated Effort: 8-10 hours ✅ (as per Issue #107)

Reviewer Notes:

• Please verify exponential backoff calculation logic
• Check jitter implementation for correctness
• Confirm logging levels are appropriate
• Review DI registration lifetime scopes
• Validate configuration structure
• Ensure documentation clarity and completeness