feat: Add Graceful Shutdown Handling to ProcessWorker

[artcava]

📋 Description

Implements comprehensive graceful shutdown handling in ProcessWorker to ensure clean termination without message loss, proper resource cleanup, and coordinated shutdown with the host application.

This PR addresses issue #100 and implements Phase 7.1 requirements for graceful shutdown in the Process Engine.

🎯 Changes

Core Implementation

• ✅ Add ConcurrentDictionary<string, Task> to track active messages
• ✅ Expose IsShuttingDown and ActiveMessageCount public properties
• ✅ Implement wait-for-completion logic with 30s timeout
• ✅ Reject new messages during shutdown (NACK with requeue)
• ✅ Use fresh CancellationToken for error recording operations
• ✅ Add comprehensive shutdown logging at all stages
• ✅ Record cancellation errors for interrupted processes

Configuration

• ✅ Configure host shutdown timeout to 45 seconds (30s + 15s buffer)
• ✅ Register ProcessWorker as singleton for health check injection
• ✅ Add health checks with proper status reporting

Health Check

• ✅ Implement ProcessWorkerHealthCheck for monitoring
• ✅ Report Healthy during normal operation
• ✅ Report Degraded when shutting down or high message count (>100)
• ✅ Include activeMessages count in response data

Testing

• ✅ Add ProcessWorkerShutdownTests for shutdown scenarios
• ✅ Add ProcessWorkerHealthCheckTests for health check validation
• ✅ All tests passing

Documentation

• ✅ Create comprehensive GRACEFUL-SHUTDOWN.md guide
• ✅ Document two-timeout strategy and rationale
• ✅ Provide testing instructions (unit, local, Docker, Kubernetes)
• ✅ Include Kubernetes integration examples
• ✅ Document fresh CancellationToken pattern
• ✅ Add troubleshooting scenarios

🔗 Related Issues

Closes #100

📝 Type of Change

• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

✅ Testing Performed

Unit Tests

✅ ProcessWorkerShutdownTests - All tests passing
✅ ProcessWorkerHealthCheckTests - All tests passing
✅ Existing ProcessWorkerTests - All tests passing

Build Verification

✅ dotnet build - Success
✅ dotnet test - All tests passing

Code Quality

• ✅ Follows CODING-CONVENTIONS.md
• ✅ XML documentation on all public members
• ✅ Structured logging with named placeholders
• ✅ Proper null checking and exception handling
• ✅ CancellationToken properly propagated

📚 Documentation

• Code is self-documenting with clear naming
• XML documentation comments added
• Comprehensive guide in docs/GRACEFUL-SHUTDOWN.md
• Testing instructions provided
• Kubernetes integration examples included

🎨 Implementation Highlights

Two-Timeout Strategy

Worker Timeout (30s):

• Internal timeout for active message completion
• Allows graceful handling of in-progress operations
• Logs warnings for stragglers

Host Timeout (45s):

• External timeout for entire application
• Includes worker shutdown + cleanup + 15s buffer
• Prevents indefinite hangs

Message Requeue Strategy

Messages cancelled during shutdown are:

1. NACK'd with requeue=true → Processed after restart
2. Marked with error → PROCESS_CANCELLED with retryable: true
3. Recorded in audit trail → Client can query process status

Benefits: Zero message loss, eventual consistency, clear audit trail

Fresh CancellationToken Pattern

During shutdown, critical operations (error recording) use a fresh CancellationTokenSource with short timeout (5s) to ensure completion even when main token is cancelled.

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
await _processService.RecordProcessErrorAsync(..., cts.Token);

🚀 Deployment Notes

Kubernetes Integration

The health check enables clean pod termination:

1. Health check returns Degraded during shutdown
2. Kubernetes stops routing new traffic
3. In-flight messages complete within timeout
4. Pod terminates cleanly

Monitoring Recommendations

Key Metrics:

• Shutdown duration (alert if >20s)
• Timeout exceeded count (indicates slow handlers)
• Message requeue rate (indicates frequent restarts)
• Active message count (alert if >100)

📋 Checklist

• My code follows the style guidelines (CODING-CONVENTIONS.md)
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published

🔍 Review Focus Areas

1. Shutdown Logic: Verify timeout handling and message tracking
2. Fresh CancellationToken Pattern: Ensure error recording works during shutdown
3. Health Check Integration: Validate status reporting
4. Test Coverage: Review shutdown scenarios
5. Documentation: Check clarity and completeness of GRACEFUL-SHUTDOWN.md

📊 Impact Assessment

Risk Level: Low

• No breaking changes to existing APIs
• Backward compatible with current ProcessWorker usage
• Only adds new functionality for graceful shutdown

Performance Impact: Negligible

• ConcurrentDictionary overhead is minimal
• Tracking only active messages (typically <100)
• No impact during normal operation

🎓 References

• Issue #100
• TECHNICAL-ANALYSIS.md - Phase 7.1
• .NET Generic Host Shutdown
• Graceful Shutdown Best Practices

---

Ready for review ✅