PowerShell S3 File Sync Watcher

Real-time file synchronization from Windows local folders to AWS S3 with CloudWatch logging. This PowerShell script uses the FSWatcherEngineEvent module to monitor file system changes and automatically upload files to S3.

Features

• Real-time Monitoring: Automatically detects file and folder changes using FileSystemWatcher
• Multi-Folder Support: Monitors multiple local directories simultaneously
• AWS S3 Integration: Seamlessly syncs files to specified S3 buckets and prefixes
• CloudWatch Logging: Comprehensive logging to AWS CloudWatch for audit and troubleshooting
• Duplicate Event Filtering: Intelligent deduplication prevents redundant uploads
• Recursive Subdirectory Support: Monitors all subdirectories within watched folders
• File and Folder Support: Handles both individual file uploads and folder syncs

Prerequisites

Required Software:

• Windows PowerShell 5.1+ or PowerShell Core 7+
• AWS CLI installed and configured\
• AWS PowerShell Module (AWSPowerShell)
• FSWatcherEngineEvent Module

AWS Requirements

• AWS Account with appropriate permissions
• S3 bucket created
• IAM user/role with permissions for:
  • s3:PutObject
  • s3:ListBucket
  • logs:CreateLogGroup
  • logs:CreateLogStream
  • logs:PutLogEvents

##Installation

1. Install Required PowerShell Modules

# Install FSWatcherEngineEvent module
Install-Module -Name FSWatcherEngineEvent -Force

# Install AWS PowerShell module
Install-Module -Name AWSPowerShell -Force

# Verify installations
Get-Module -ListAvailable -Name FSWatcherEngineEvent
Get-Module -ListAvailable -Name AWSPowerShell

2. Install and Configure AWS CLI

# Download AWS CLI from: https://aws.amazon.com/cli/

# Configure AWS credentials
aws configure

Provide your:

• AWS Access Key ID
• WS Secret Access Key
• Default region (e.g., us-west-1)
• Default output format (e.g., json)

3. Clone This Repository

git clone https://github.com/yourusername/powershell-s3-filesync-watcher.git
cd powershell-s3-filesync-watcher

Configuration

Edit the script variables to match your environment:

# AWS Configuration
$AWSRegion       = "us-west-1"              # Your AWS region
$s3BucketName    = "your-s3-bucket-name"   # Your S3 bucket

# Local Folder Paths (must exist)
$localFileExchangePath = "D:\home\ftp\FileExchange"
$localFtpFilesPath     = "D:\home\ftp\ftpFiles"

# S3 Prefixes (folder structure in S3)
$s3FileExchangePrefix = "FileExchange/"
$s3FtpFilesPrefix     = "ftpFiles/"

# CloudWatch Logging
$logGroupName   = "/your/log/group"
$logStreamName  = "your-log-stream"

Folder Structure Example

Local:

D:\home\ftp\
├── FileExchange\
│   ├── file1.txt
│   └── subfolder\
│       └── file2.txt
└── ftpFiles\
    └── data.csv

S3 Result:

s3://your-bucket/
├── FileExchange/
│   ├── file1.txt
│   └── subfolder/
│       └── file2.txt
└── ftpFiles/
    └── data.csv

Usage

Run the Script

# Navigate to script directory
cd powershell-s3-filesync-watcher

# Run with administrator privileges (recommended)
.\S3-FileSyncWatcher.ps1

Expected Output

Initializing CloudWatch Log Group and Log Stream...
Setting up watcher for FileExchange folder: D:\home\ftp\FileExchange
Setting up watcher for ftpFiles folder: D:\home\ftp\ftpFiles
Monitoring file changes. Press Ctrl+C to stop.

Detected change: Created -> D:\home\ftp\FileExchange\newfile.txt (FileExchange)
Uploading file 'D:\home\ftp\FileExchange\newfile.txt' to 's3://your-bucket/FileExchange/newfile.txt'...
Upload completed for 'D:\home\ftp\FileExchange\newfile.txt'.

###Stop Monitoring Press Ctrl+C in the PowerShell window.

##How It Works

• Initialization: Script creates CloudWatch log group and stream if they don't exist
• FileSystemWatcher Setup: Two watchers monitor the configured local folders
• Event Detection: When files/folders are created, modified, or deleted:
  • Event is logged to CloudWatch
  • Duplicate events within 5 seconds are filtered
  • Changed path is calculated relative to the base folder
• S3 Synchronization:
  • Files: Individual aws s3 cp command
  • Folders: Recursive aws s3 sync command
• Continuous Monitoring: Script runs indefinitely until manually stopped

##Monitoring

###Check CloudWatch Logs

# View logs in AWS Console
# Navigate to: CloudWatch > Log Groups > /your/log/group > your-log-stream

# Or use AWS CLI
aws logs tail /your/log/group --follow --region us-west-1

Verify S3 Upload

# List files in S3
aws s3 ls s3://your-bucket/FileExchange/ --recursive --region us-west-1