# Production Monitoring

Best practices for deploying and monitoring ChronoTrace in production environments, including performance optimization, security considerations, and operational procedures.

---

## 🏭 Production Deployment Strategy

### 🎯 Production Goals

- **Minimal Performance Impact** (<1% overhead)
- **High Reliability** (99.9% uptime)
- **Security Compliance** (PII protection, audit trails)
- **Operational Efficiency** (automated monitoring, alerting)
- **Cost Effectiveness** (optimized storage, retention)

---

## ⚙️ Production Configuration

### Essential Production Settings

```bash
# .env - Production optimized configuration
CHRONOTRACE_ENABLED=true
CHRONOTRACE_MODE=record_on_error
CHRONOTRACE_SAMPLE_RATE=0.001  # 0.1% of successful requests

# Performance optimizations
CHRONOTRACE_ASYNC_STORAGE=true
CHRONOTRACE_QUEUE_CONNECTION=redis
CHRONOTRACE_QUEUE_NAME=chronotrace

# Storage optimization
CHRONOTRACE_STORAGE=s3
CHRONOTRACE_RETENTION_DAYS=7
CHRONOTRACE_AUTO_PURGE=true

# Security
CHRONOTRACE_SCRUB_PII=true
CHRONOTRACE_CAPTURE_CACHE=false
CHRONOTRACE_CAPTURE_EVENTS=false
```

### Comprehensive Production Config

```php
// config/chronotrace.php - Production optimized
return [
    'enabled' => env('CHRONOTRACE_ENABLED', true),
    'mode' => env('CHRONOTRACE_MODE', 'record_on_error'),
    
    // Error-focused recording
    'sample_rate' => env('CHRONOTRACE_SAMPLE_RATE', 0.001),
    'error_conditions' => [
        'status_codes' => [400, 401, 403, 404, 422, 500, 502, 503, 504],
        'slow_requests' => 5000,  // >5 seconds
        'memory_threshold' => 256 * 1024 * 1024,  // >256MB
    ],
    
    // Performance optimizations
    'async_storage' => true,
    'queue_connection' => 'redis',
    'compression' => [
        'enabled' => true,
        'level' => 6,
    ],
    
    // Event capture - minimal for production
    'capture' => [
        'database' => true,   // Essential for debugging
        'http' => true,       // Critical for API issues
        'jobs' => true,       // Important for async work
        'cache' => false,     // Usually noisy
        'events' => false,    // Very verbose
    ],
    
    // Security and compliance
    'scrub' => [
        'password', 'token', 'secret', 'key',
        'email', 'phone', 'ssn', 'credit_card',
        'api_key', 'auth_token', 'bearer_token',
    ],
    
    'excluded_routes' => [
        'password/*',
        'api/auth/*',
        'api/payments/sensitive/*',
        'admin/secrets/*',
    ],
    
    // Storage and retention
    'storage' => env('CHRONOTRACE_STORAGE', 's3'),
    'retention_days' => env('CHRONOTRACE_RETENTION_DAYS', 7),
    'auto_purge' => true,
    
    // Monitoring and alerting
    'monitoring' => [
        'error_threshold' => 5,        // Alert after 5 errors in 5 minutes
        'performance_threshold' => 10, // Alert if >10 slow requests in 5 minutes
        'storage_threshold' => 0.8,    // Alert at 80% storage capacity
    ],
];
```

---

## 🚀 Deployment Process

### 1. Pre-Deployment Checklist

```bash
# Infrastructure readiness
- [ ] Redis cluster configured and tested
- [ ] S3 bucket created with proper permissions
- [ ] Queue workers scaled appropriately
- [ ] Monitoring dashboards prepared
- [ ] Alerting rules configured

# Security validation
- [ ] PII scrubbing tested
- [ ] Sensitive routes excluded
- [ ] Access controls verified
- [ ] Audit logging enabled

# Performance validation
- [ ] Load testing with ChronoTrace enabled
- [ ] Memory usage profiled
- [ ] Storage growth estimated
- [ ] Queue processing capacity verified
```

### 2. Gradual Rollout Strategy

#### Phase 1: Limited Deployment (1% of traffic)
```bash
# Deploy to single instance
CHRONOTRACE_ENABLED=true
CHRONOTRACE_MODE=record_on_error
CHRONOTRACE_SAMPLE_RATE=0.001

# Monitor for 24 hours
- Check performance impact
- Verify storage usage
- Monitor error capture rate
- Validate queue processing
```

#### Phase 2: Expanded Deployment (10% of traffic)
```bash
# Scale to more instances
# Monitor for 48 hours
- Performance metrics stable
- No memory leaks detected
- Storage usage predictable
- Queue workers handling load
```

#### Phase 3: Full Production (100% of traffic)
```bash
# Deploy to all instances
# Continuous monitoring
- Set up automated alerts
- Implement health checks
- Monitor storage growth
- Track performance trends
```

---

## 📊 Production Monitoring

### 1. Performance Monitoring

#### Application Performance Metrics
```php
// Custom metrics collection for production
class ChronoTraceMetrics
{
    public function collectMetrics()
    {
        return [
            'traces_per_hour' => $this->getTracesPerHour(),
            'average_trace_size' => $this->getAverageTraceSize(),
            'error_capture_rate' => $this->getErrorCaptureRate(),
            'storage_usage_mb' => $this->getStorageUsage(),
            'queue_processing_time' => $this->getQueueProcessingTime(),
            'memory_overhead_mb' => $this->getMemoryOverhead(),
        ];
    }
    
    private function getTracesPerHour()
    {
        return ChronoTrace::where('created_at', '>=', now()->subHour())->count();
    }
    
    private function getErrorCaptureRate()
    {
        $totalErrors = Log::where('level', 'error')
            ->where('created_at', '>=', now()->subHour())
            ->count();
            
        $capturedErrors = ChronoTrace::where('status', '>=', 400)
            ->where('created_at', '>=', now()->subHour())
            ->count();
            
        return $totalErrors > 0 ? ($capturedErrors / $totalErrors) * 100 : 0;
    }
}
```

#### Real-time Performance Dashboard
```bash
# Monitor key production metrics
php artisan chronotrace:monitor --dashboard

# Expected output:
┌─ CHRONOTRACE PRODUCTION DASHBOARD ──────────────────────────┐
│ Status: 🟢 Operational                                      │
│ Mode: record_on_error (0.1% sampling)                      │
│                                                             │
│ 📊 Last Hour Metrics:                                      │
│   Traces Captured: 156                                     │
│   Error Rate: 2.3% (4 errors out of 173 requests)         │
│   Avg Trace Size: 45KB                                     │
│   Storage Used: 1.2GB (15% of quota)                       │
│                                                             │
│ ⚡ Performance:                                             │
│   Queue Processing: 234ms avg                              │
│   Memory Overhead: 12MB avg                                │
│   Request Overhead: 0.8ms avg                              │
│                                                             │
│ 🚨 Alerts: None active                                     │
└─────────────────────────────────────────────────────────────┘
```

### 2. Health Checks

```php
// Production health check endpoint
class ChronoTraceHealthCheck
{
    public function check()
    {
        $health = [
            'status' => 'healthy',
            'checks' => []
        ];
        
        // Check if ChronoTrace is enabled and functional
        $health['checks']['enabled'] = config('chronotrace.enabled');
        
        // Check storage accessibility
        try {
            Storage::disk('chronotrace')->put('health_check.txt', 'ok');
            Storage::disk('chronotrace')->delete('health_check.txt');
            $health['checks']['storage'] = 'accessible';
        } catch (Exception $e) {
            $health['checks']['storage'] = 'failed';
            $health['status'] = 'unhealthy';
        }
        
        // Check queue processing
        $health['checks']['queue'] = $this->checkQueueHealth();
        
        // Check recent error capture
        $recentTraces = ChronoTrace::where('created_at', '>=', now()->subMinutes(5))->count();
        $health['checks']['recent_activity'] = $recentTraces > 0 ? 'active' : 'quiet';
        
        return $health;
    }
}
```

### 3. Automated Alerting

#### Critical Alerts Configuration
```php
// config/chronotrace-alerts.php
return [
    'alerts' => [
        'high_error_rate' => [
            'threshold' => 10,  // >10 errors in 5 minutes
            'window' => 300,    // 5 minutes
            'channels' => ['slack', 'pagerduty'],
            'message' => 'ChronoTrace: High error rate detected'
        ],
        
        'queue_backlog' => [
            'threshold' => 1000, // >1000 jobs in queue
            'channels' => ['slack'],
            'message' => 'ChronoTrace: Queue backlog building up'
        ],
        
        'storage_full' => [
            'threshold' => 85,   // >85% storage used
            'channels' => ['slack', 'email'],
            'message' => 'ChronoTrace: Storage approaching limit'
        ],
        
        'service_down' => [
            'threshold' => 0,    // No traces in 10 minutes
            'window' => 600,
            'channels' => ['pagerduty'],
            'message' => 'ChronoTrace: Service appears down'
        ]
    ]
];
```

#### Alert Implementation
```php
class ChronoTraceAlerting
{
    public function checkAlerts()
    {
        foreach (config('chronotrace-alerts.alerts') as $name => $config) {
            switch ($name) {
                case 'high_error_rate':
                    $this->checkHighErrorRate($config);
                    break;
                case 'queue_backlog':
                    $this->checkQueueBacklog($config);
                    break;
                case 'storage_full':
                    $this->checkStorageUsage($config);
                    break;
                case 'service_down':
                    $this->checkServiceHealth($config);
                    break;
            }
        }
    }
    
    private function checkHighErrorRate($config)
    {
        $errorCount = ChronoTrace::where('status', '>=', 400)
            ->where('created_at', '>=', now()->subSeconds($config['window']))
            ->count();
            
        if ($errorCount > $config['threshold']) {
            $this->sendAlert($config, "Error rate: {$errorCount} errors in {$config['window']} seconds");
        }
    }
}
```

---

## 🔧 Queue Management

### Production Queue Configuration

```php
// config/queue.php - Production queue setup
'connections' => [
    'chronotrace' => [
        'driver' => 'redis',
        'connection' => 'chronotrace',
        'queue' => env('CHRONOTRACE_QUEUE_NAME', 'chronotrace'),
        'retry_after' => 90,
        'block_for' => null,
    ],
],

// config/database.php - Dedicated Redis for ChronoTrace
'redis' => [
    'chronotrace' => [
        'host' => env('CHRONOTRACE_REDIS_HOST', '127.0.0.1'),
        'password' => env('CHRONOTRACE_REDIS_PASSWORD', null),
        'port' => env('CHRONOTRACE_REDIS_PORT', 6379),
        'database' => env('CHRONOTRACE_REDIS_DATABASE', 2),
    ],
],
```

### Queue Worker Management

```bash
# Production queue worker configuration
# /etc/supervisor/conf.d/chronotrace-worker.conf

[program:chronotrace-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/artisan queue:work redis --queue=chronotrace --sleep=3 --tries=3 --max-time=3600 --timeout=30
directory=/path/to/project
autostart=true
autorestart=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/log/chronotrace-worker.log
stopwaitsecs=3600
```

### Queue Monitoring Commands

```bash
# Monitor queue health
php artisan chronotrace:queue-status

# Expected output:
┌─ CHRONOTRACE QUEUE STATUS ──────────────────────────────────┐
│ Connection: redis (chronotrace)                             │
│ Queue: chronotrace                                          │
│                                                             │
│ 📋 Current Status:                                         │
│   Pending Jobs: 23                                         │
│   Failed Jobs: 2                                           │
│   Processed (last hour): 1,456                            │
│   Average Processing Time: 234ms                           │
│                                                             │
│ 👥 Workers:                                                │
│   Active Workers: 4                                        │
│   Worker Status: All healthy                               │
│   Last Processed: 2 seconds ago                            │
│                                                             │
│ ⚠️  Issues: None detected                                  │
└─────────────────────────────────────────────────────────────┘

# Restart queue workers if needed
sudo supervisorctl restart chronotrace-worker:*

# Monitor failed jobs
php artisan queue:failed --queue=chronotrace
```

---

## 💾 Storage Management

### S3 Production Configuration

```bash
# S3 environment variables
CHRONOTRACE_STORAGE=s3
CHRONOTRACE_S3_BUCKET=production-chronotrace
CHRONOTRACE_S3_REGION=us-east-1
CHRONOTRACE_S3_PREFIX=traces/production
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...

# S3 bucket policy for cost optimization
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {"AWS": "arn:aws:iam::ACCOUNT:user/chronotrace"},
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::production-chronotrace/*"
        }
    ]
}
```

### Storage Optimization

```php
// Implement intelligent retention policies
class ProductionStorageManager
{
    public function optimizeStorage()
    {
        // Different retention for different trace types
        $this->cleanupOldTraces([
            'success_traces' => 3,   // 3 days for successful requests
            'error_traces' => 14,    // 14 days for errors
            'critical_errors' => 30, // 30 days for 5xx errors
        ]);
        
        // Compress old traces
        $this->compressOldTraces(7); // Compress traces older than 7 days
        
        // Archive to cheaper storage
        $this->archiveToGlacier(30); // Archive traces older than 30 days
    }
    
    private function cleanupOldTraces($retentionPolicies)
    {
        foreach ($retentionPolicies as $type => $days) {
            $count = 0;
            
            switch ($type) {
                case 'success_traces':
                    $count = $this->deleteOldTraces(['status' => 200], $days);
                    break;
                case 'error_traces':
                    $count = $this->deleteOldTraces(['status' => [400, 499]], $days);
                    break;
                case 'critical_errors':
                    $count = $this->deleteOldTraces(['status' => [500, 599]], $days);
                    break;
            }
            
            Log::info("Storage cleanup: {$type}", ['deleted' => $count, 'retention_days' => $days]);
        }
    }
}
```

### Storage Monitoring

```bash
# Monitor storage usage
php artisan chronotrace:storage-stats

# Expected output:
┌─ CHRONOTRACE STORAGE STATISTICS ────────────────────────────┐
│ Storage Driver: S3 (production-chronotrace)                │
│ Region: us-east-1                                           │
│                                                             │
│ 📊 Usage Statistics:                                       │
│   Total Traces: 45,678                                     │
│   Total Size: 2.3GB                                        │
│   Average Trace Size: 52KB                                 │
│                                                             │
│ 📅 Retention Breakdown:                                    │
│   Last 24h: 1,234 traces (67MB)                           │
│   Last 7 days: 8,901 traces (456MB)                       │
│   Last 30 days: 34,567 traces (1.8GB)                     │
│                                                             │
│ 💰 Cost Estimates:                                         │
│   Monthly Storage: $12.50                                  │
│   Monthly Requests: $3.20                                  │
│   Total Monthly: $15.70                                    │
│                                                             │
│ 🧹 Next Cleanup: 2024-08-07 02:00 UTC                     │
└─────────────────────────────────────────────────────────────┘
```

---

## 🔒 Security in Production

### Access Control

```php
// Implement role-based access to traces
class ChronoTraceAccessControl
{
    public function canViewTrace(User $user, $traceId)
    {
        // Only allow access to own traces for regular users
        if (!$user->hasRole(['admin', 'developer'])) {
            $trace = ChronoTrace::find($traceId);
            return $trace && $trace->user_id === $user->id;
        }
        
        // Admins and developers can view all traces
        return true;
    }
    
    public function canDeleteTrace(User $user, $traceId)
    {
        // Only admins can delete traces
        return $user->hasRole('admin');
    }
}
```

### Audit Logging

```php
// Track access to sensitive traces
class ChronoTraceAuditLogger
{
    public function logTraceAccess($user, $traceId, $action)
    {
        Log::channel('audit')->info('ChronoTrace access', [
            'user_id' => $user->id,
            'user_email' => $user->email,
            'trace_id' => $traceId,
            'action' => $action, // view, replay, delete
            'ip_address' => request()->ip(),
            'user_agent' => request()->userAgent(),
            'timestamp' => now()->toISOString(),
        ]);
    }
}
```

---

## 📈 Performance Optimization

### Production Performance Tuning

```php
// Optimize for production workloads
class ProductionOptimizer
{
    public function optimizeForProduction()
    {
        // Reduce memory usage
        $this->optimizeMemoryUsage();
        
        // Optimize database queries
        $this->optimizeDatabaseQueries();
        
        // Implement caching
        $this->implementCaching();
        
        // Optimize queue processing
        $this->optimizeQueueProcessing();
    }
    
    private function optimizeMemoryUsage()
    {
        // Limit trace payload size
        config([
            'chronotrace.memory.max_trace_size' => 2 * 1024 * 1024, // 2MB max
            'chronotrace.compression.enabled' => true,
            'chronotrace.compression.level' => 9, // Maximum compression
        ]);
    }
    
    private function optimizeQueueProcessing()
    {
        // Batch processing for better throughput
        config([
            'chronotrace.queue.batch_size' => 50,
            'chronotrace.queue.batch_timeout' => 5, // seconds
        ]);
    }
}
```

### Load Testing

```bash
# Load test with ChronoTrace enabled
# test-chronotrace-performance.sh

#!/bin/bash

echo "Starting ChronoTrace load test..."

# Enable ChronoTrace
php artisan chronotrace:record --duration=30m &

# Run load test
ab -n 10000 -c 100 -t 1800 http://localhost:8000/api/test &
wrk -t12 -c400 -d30m http://localhost:8000/api/test &

# Monitor performance
while true; do
    echo "=== $(date) ==="
    php artisan chronotrace:monitor --brief
    sleep 60
done
```

---

## 📋 Production Checklist

### Daily Operations
- [ ] Check queue worker status
- [ ] Monitor error rates
- [ ] Review storage usage
- [ ] Check alert notifications
- [ ] Verify backup status

### Weekly Operations
- [ ] Review performance trends
- [ ] Analyze error patterns
- [ ] Update retention policies
- [ ] Security audit review
- [ ] Cost optimization review

### Monthly Operations
- [ ] Full performance review
- [ ] Capacity planning
- [ ] Security assessment
- [ ] Documentation updates
- [ ] Team training review

---

## 🚨 Incident Response

### ChronoTrace-Related Incidents

#### High Queue Backlog
```bash
# Immediate response
1. Check queue worker status: supervisorctl status chronotrace-worker:*
2. Scale workers if needed: supervisorctl start chronotrace-worker:*
3. Monitor queue depth: watch "php artisan queue:monitor chronotrace"
4. If critical, temporarily disable: CHRONOTRACE_ENABLED=false
```

#### Storage Issues
```bash
# Storage full response
1. Check storage usage: php artisan chronotrace:storage-stats
2. Emergency cleanup: php artisan chronotrace:purge --days=1 --force
3. Scale storage if needed
4. Review retention policies
```

#### Performance Impact
```bash
# High overhead response
1. Check current mode: php artisan chronotrace:diagnose
2. Reduce sampling: CHRONOTRACE_SAMPLE_RATE=0.0001
3. Disable non-critical events: CHRONOTRACE_CAPTURE_CACHE=false
4. Monitor improvement: php artisan chronotrace:monitor
```

---

## 📚 Related Documentation

- **[Configuration](Configuration.md)** - Detailed configuration options
- **[Security](Security.md)** - Security best practices
- **[Commands](Commands.md)** - Production management commands
- **[Troubleshooting](Troubleshooting.md)** - Common production issues

---

**Production deployment success!** ChronoTrace is now monitoring your application with minimal overhead while providing powerful debugging capabilities when you need them most.