File Monitoring & Publishing Service

A Spring Boot application that monitors S3-compatible storage (MinIO), caches files in Redis/Valkey, and publishes file metadata to RabbitMQ when files are ready for processing.

Features

✅ S3/MinIO Integration - Monitor and sync files from MinIO (S3-compatible object storage)
✅ Redis Caching - Cache file content and metadata in Valkey/Redis for fast retrieval
✅ Smart Hash Tracking - Only publish messages when file content actually changes
✅ RabbitMQ Publishing - Publish file metadata to RabbitMQ for downstream processing
✅ Periodic Synchronization - Automatic sync every 30 seconds with configurable intervals
✅ Multi-Bucket Support - Monitor and sync multiple S3 buckets simultaneously
✅ REST API - Browse cache, trigger syncs, and manage file operations

Architecture

MinIO (S3)
    ↓
S3FileService (Optimized with HeadObject, Pagination)
    ↓
SyncService (30-sec periodic sync)
    ↓
CacheService (Redis storage)
    ↓
RabbitMQPublisher (Hash-based change detection)
    ↓
RabbitMQ (ready-for-parsing queue)

API Endpoints

File Download

• GET /download/{bucketName}/{filename} - Download file from cache (if available)
• GET /list - List all files from all buckets
• GET /list/{bucketName} - List files in specific bucket
• GET /buckets - List all available buckets

Cache Management

• GET /cache/files - View all cached files with pagination
• GET /cache/stats - Get cache statistics (total files, memory usage)
• GET /cache/bucket/{bucketName} - View cached files for specific bucket

Sync Management

• GET /sync/trigger - Manually trigger S3 to cache sync
• GET /sync/clear/{bucketName} - Clear all cached files for a bucket