Optimize Local Disk Usage in BrowserState

[bigboateng]

Currently, when BrowserState mounts a storage provider (local, S3, GCS, Redis), it creates local copies of files on disk. This approach ensures fast access to browser profiles, but without proper cleanup mechanisms, it can lead to excessive disk usage over time, especially in high-volume automation environments.

Current Behavior

• All storage providers create local copies during the mount operation
• While temporary files are cleaned up during unmount (with autoCleanup: true), the main storage directory continues to grow
• No built-in mechanism to limit or rotate storage based on age, size, or usage patterns

Proposed Solution

We should implement disk usage optimization mechanisms, such as:

1. Size-based limits: Allow users to specify maximum local storage size (e.g., 10GB) and automatically clean up oldest profiles when the limit is reached.

2. Time-based cleanup: Implement auto-expiry or TTL (Time To Live) for locally cached profiles. Profiles not accessed for X days could be removed from local storage.

3. Usage-based rotation: Maintain a LRU (Least Recently Used) cache for profiles, keeping only the most frequently accessed ones locally.

4. Disk usage metrics: Add methods to check current disk usage and estimate growth patterns.

5. Selective syncing: Allow users to specify which parts of profiles to sync (e.g., cookies only, no cache files, etc.)

Implementation Ideas

• Add config options to BrowserState constructor:

  {
    diskLimit: '10GB',
    localTTL: '7d', // Remove locally cached profiles after 7 days
    maxProfiles: 50, // Keep only 50 most recent profiles
    diskUsageWarningThreshold: '80%' // Warn when disk usage exceeds threshold
  }

• Add utility methods:

  // Get current disk usage stats
  await browserState.getDiskUsageStats();
  
  // Manually clean up old profiles
  await browserState.cleanupOldProfiles(options);

Benefits

• Reduced risk of disk full errors in production environments
• Better resource utilization
• Lower operational costs for cloud-based automation systems
• Improved reliability for long-running services

Questions to Resolve

1. How to handle conflicting cases where a profile is old but frequently used?
2. Should we implement different strategies for different storage providers?
3. What default values would be appropriate for most use cases?

Related Areas

• Profile compression strategies
• Storage pruning logic
• Metrics and monitoring integration

Priority

Medium - This is important for production scalability but not a critical bug.