Storage

Storage

ChronoTrace supports multiple storage backends for storing trace data. This guide covers configuration and management of different storage options.

Storage Backends

Local Storage (Default)

Configuration:

'storage' => 'local',
'path' => storage_path('chronotrace'),

Pros:

• Simple setup, no external dependencies
• Fast access for development
• No additional costs

Cons:

• Not scalable across multiple servers
• Limited by local disk space
• No built-in backup/replication

Best for: Development and single-server deployments

Cloud Storage (S3/MinIO)

Configuration:

'storage' => 's3',
's3' => [
    'bucket' => 'my-chronotrace-bucket',
    'region' => 'us-east-1',
    'path_prefix' => 'traces',
],

Pros:

• Highly scalable and reliable
• Built-in backup and versioning
• Cost-effective for large volumes
• Multi-region support

Cons:

• Network latency for access
• Requires AWS/cloud setup
• Additional costs

Best for: Production deployments and multi-server setups

---

Storage Configuration

Local Storage Setup

// config/chronotrace.php
'storage' => 'local',
'local' => [
    'path' => storage_path('chronotrace'),
    'create_directories' => true,
    'permissions' => [
        'file' => 0644,
        'dir' => 0755,
    ],
],

Directory Structure

storage/chronotrace/
├── traces/           # Individual trace files
├── indexes/          # Search indexes for fast lookup
├── temp/            # Temporary files during processing
└── metadata/        # Trace metadata and statistics

Permissions Setup

# Set proper permissions
chmod -R 755 storage/chronotrace
chown -R www-data:www-data storage/chronotrace

# Verify permissions
ls -la storage/chronotrace/

S3 Storage Setup

For detailed S3 configuration, see the S3 & MinIO Storage guide.

// config/chronotrace.php
'storage' => 's3',
's3' => [
    'bucket' => env('CHRONOTRACE_S3_BUCKET'),
    'region' => env('CHRONOTRACE_S3_REGION', 'us-east-1'),
    'path_prefix' => env('CHRONOTRACE_S3_PREFIX', 'traces'),
    'storage_class' => 'STANDARD_IA',
    'server_side_encryption' => 'AES256',
],

Environment Variables

# .env
CHRONOTRACE_STORAGE=s3
CHRONOTRACE_S3_BUCKET=my-app-chronotrace
CHRONOTRACE_S3_REGION=us-east-1
CHRONOTRACE_S3_PREFIX=production/traces
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key

---

Storage Optimization

Compression

Enable compression to reduce storage usage:

// config/chronotrace.php
'compression' => [
    'enabled' => true,
    'level' => 6,              // 1-9, higher = better compression
    'threshold' => 1024,       // Only compress files > 1KB
    'format' => 'gzip',        // gzip or brotli
],

Compression benefits:

• 60-80% size reduction typically
• Lower storage costs
• Faster transfers (for S3)

Retention Policies

Configure automatic cleanup:

// config/chronotrace.php
'retention_days' => 15,
'auto_purge' => true,
'purge_schedule' => 'daily',

'retention_policies' => [
    'error_traces' => 30,      // Keep errors longer
    'slow_traces' => 21,       // Keep slow requests longer
    'success_traces' => 7,     // Clean up successful requests sooner
],

Storage Partitioning

Organize traces efficiently:

// config/chronotrace.php
'partitioning' => [
    'strategy' => 'date',      // date, size, hash
    'date_format' => 'Y/m/d',  // Creates YYYY/MM/DD structure
    'max_files_per_directory' => 1000,
],

Results in structure like:

traces/
├── 2024/08/06/
│   ├── trace_001.json
│   ├── trace_002.json
│   └── ...
├── 2024/08/07/
│   └── ...

---

Storage Monitoring

Storage Usage Commands

# Check storage statistics
php artisan chronotrace:storage-stats

# Expected output:
┌─ STORAGE STATISTICS ────────────────────────────────────────┐
│ Storage Type: Local                                         │
│ Location: /app/storage/chronotrace                         │
│                                                             │
│ 📊 Usage:                                                  │
│   Total Traces: 1,234                                      │
│   Total Size: 156 MB                                       │
│   Average Size: 126 KB                                     │
│   Compressed: 89% reduction                                │
│                                                             │
│ 📅 Age Distribution:                                       │
│   Last 24h: 89 traces (12 MB)                             │
│   Last 7 days: 567 traces (78 MB)                         │
│   Older: 578 traces (66 MB)                               │
│                                                             │
│ 💾 Available Space: 2.1 GB                                │
└─────────────────────────────────────────────────────────────┘

Automated Monitoring

// Monitor storage health
class StorageHealthMonitor
{
    public function checkStorageHealth()
    {
        $stats = [
            'total_size' => $this->getTotalStorageSize(),
            'available_space' => $this->getAvailableSpace(),
            'trace_count' => $this->getTraceCount(),
            'average_size' => $this->getAverageTraceSize(),
        ];
        
        // Alert if storage > 80% full
        if ($stats['total_size'] / $this->getTotalSpace() > 0.8) {
            $this->sendStorageAlert($stats);
        }
        
        return $stats;
    }
}

---

Storage Migration

Local to S3 Migration

# Migration script
php artisan chronotrace:migrate-storage --from=local --to=s3 --verify

# Expected output:
Migrating traces from local to S3...
✓ Migrated trace_001.json (125 KB)
✓ Migrated trace_002.json (89 KB)
✓ Migrated trace_003.json (156 KB)
...
✓ Migration complete: 1,234 traces migrated (156 MB total)
✓ Verification: All traces accessible in S3
✓ Local cleanup: 156 MB freed

Migration Strategy

// config/chronotrace.php
'migration' => [
    'batch_size' => 100,       // Process 100 traces at a time
    'verify_uploads' => true,   // Verify each upload
    'cleanup_source' => true,   // Remove from source after successful migration
    'preserve_metadata' => true, // Keep original timestamps
],

---

Custom Storage Drivers

Creating Custom Driver

// app/Storage/CustomStorageDriver.php
class CustomStorageDriver implements StorageDriverInterface
{
    public function store(string $traceId, array $data): bool
    {
        // Implement custom storage logic
        return $this->writeToCustomBackend($traceId, $data);
    }
    
    public function retrieve(string $traceId): ?array
    {
        // Implement custom retrieval logic
        return $this->readFromCustomBackend($traceId);
    }
    
    public function delete(string $traceId): bool
    {
        // Implement custom deletion logic
        return $this->deleteFromCustomBackend($traceId);
    }
    
    public function exists(string $traceId): bool
    {
        // Check if trace exists
        return $this->checkCustomBackend($traceId);
    }
}

Registering Custom Driver

// app/Providers/AppServiceProvider.php
public function boot()
{
    $this->app->afterResolving(StorageManager::class, function (StorageManager $manager) {
        $manager->extend('custom', function ($app, $config) {
            return new CustomStorageDriver($config);
        });
    });
}

Using Custom Driver

// config/chronotrace.php
'storage' => 'custom',
'custom' => [
    'endpoint' => 'https://my-storage-api.com',
    'api_key' => env('CUSTOM_STORAGE_API_KEY'),
    'timeout' => 30,
],

---

Storage Security

Encryption at Rest

// config/chronotrace.php
'encryption' => [
    'enabled' => true,
    'key' => env('CHRONOTRACE_ENCRYPTION_KEY'),
    'cipher' => 'AES-256-GCM',
],

Access Control

// Implement access control for trace storage
class TraceAccessControl
{
    public function canAccessTrace(User $user, string $traceId): bool
    {
        // Implement your access logic
        return $user->hasRole('admin') || 
               $this->tracebelongsToUser($user, $traceId);
    }
}

---

Storage Backup

Automated Backups

// config/chronotrace.php
'backup' => [
    'enabled' => true,
    'schedule' => 'daily',
    'retention' => 30,         // Keep backups for 30 days
    'destination' => 's3',     // Backup to S3
    'encryption' => true,
],

Manual Backup

# Create manual backup
php artisan chronotrace:backup --destination=s3://backup-bucket/chronotrace

# Restore from backup
php artisan chronotrace:restore --source=s3://backup-bucket/chronotrace/2024-08-06

---

Performance Optimization

Async Storage Operations

// Use queues for storage operations
'async_storage' => true,
'queue_connection' => 'redis',
'queue_name' => 'chronotrace-storage',

Connection Pooling

// Optimize storage connections
'storage_options' => [
    'pool_size' => 10,
    'timeout' => 30,
    'retry_attempts' => 3,
],

Caching

// Cache frequently accessed traces
'cache' => [
    'enabled' => true,
    'driver' => 'redis',
    'ttl' => 3600,             // Cache for 1 hour
    'max_size' => 100,         // Cache max 100 traces
],

---

Troubleshooting Storage Issues

Common Issues

Storage Full

# Check available space
df -h storage/chronotrace

# Clean up old traces
php artisan chronotrace:purge --days=7

# Move to S3 storage
php artisan chronotrace:migrate-storage --to=s3

Permission Issues

# Fix permissions
sudo chown -R www-data:www-data storage/chronotrace
sudo chmod -R 755 storage/chronotrace

S3 Connection Issues

# Test S3 connectivity
php artisan chronotrace:test-storage --driver=s3

# Check credentials
aws s3 ls s3://your-bucket --profile your-profile

---

📚 Related Documentation

• S3 & MinIO Storage - Detailed cloud storage setup
• Configuration - Complete configuration options
• Production Monitoring - Storage monitoring in production
• Troubleshooting - Storage troubleshooting guide

---

Choose the right storage backend for your needs! Start with local storage for development, then scale to cloud storage for production deployments.