AWS File Validator Lambda

An intermediate Java project that validates file formats (CSV/JSON) using AWS Lambda and uploads validated files to S3.

Features

• File Format Validation: Validates CSV and JSON file formats
• AWS Lambda Integration: Serverless processing of file validation
• S3 Upload: Automatically uploads validated files to S3 bucket
• AWS Credentials: Uses default profile from ~/.aws/credentials
• Structured Validation:
  • CSV: Checks headers, row consistency, and format
  • JSON: Validates JSON structure and reports element counts

Project Structure

aws-file-validator/
├── src/main/java/com/awsproject/
│   ├── lambda/
│   │   └── FileValidatorHandler.java    # Lambda function handler
│   ├── service/
│   │   └── S3Service.java               # S3 upload service
│   ├── validator/
│   │   ├── FileValidator.java           # Validator interface
│   │   ├── CsvValidator.java            # CSV validation logic
│   │   └── JsonValidator.java           # JSON validation logic
│   ├── model/
│   │   └── ValidationResult.java        # Validation result model
│   └── LocalTester.java                 # Local testing utility
├── sample-files/
│   ├── sample.csv                       # Sample CSV file
│   └── sample.json                      # Sample JSON file
├── pom.xml                              # Maven configuration
├── deploy.sh                            # Lambda deployment script
├── test-lambda.sh                       # Lambda testing script
└── README.md

Prerequisites

1. Java 11 or higher
2. Maven 3.6+
3. AWS CLI configured with credentials
4. AWS Account with permissions for Lambda and S3

Setup

1. Configure AWS Credentials

Ensure ~/.aws/credentials contains your default profile:

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

2. Build the Project

cd aws-file-validator
mvn clean package

This creates a deployable JAR at target/aws-file-validator-1.0.0.jar.

3. Create S3 Bucket

aws s3 mb s3://your-bucket-name --region us-east-1

Local Testing

Test the Lambda function locally before deploying:

# Compile the project
mvn clean compile

# Test with CSV file
mvn exec:java -Dexec.mainClass="com.awsproject.LocalTester" \
  -Dexec.args="your-bucket-name sample-files/sample.csv CSV"

# Test with JSON file
mvn exec:java -Dexec.mainClass="com.awsproject.LocalTester" \
  -Dexec.args="your-bucket-name sample-files/sample.json JSON"

Deployment

Deploy to AWS Lambda

chmod +x deploy.sh
./deploy.sh

This script:

1. Builds the project with Maven
2. Creates IAM role with necessary permissions (if needed)
3. Creates or updates the Lambda function

Test Deployed Lambda

chmod +x test-lambda.sh
./test-lambda.sh your-bucket-name

Usage

Lambda Input Format

{
  "fileName": "data.csv",
  "content": "header1,header2\nvalue1,value2",
  "fileType": "CSV",
  "bucketName": "your-bucket-name"
}

Parameters:

• fileName: Name of the file (required)
• content: File content as string (required)
• fileType: Either "CSV" or "JSON" (defaults to "JSON")
• bucketName: Target S3 bucket (required)

Lambda Output Format

Success Response:

{
  "statusCode": 200,
  "valid": true,
  "message": "File validated and uploaded successfully",
  "validationDetails": "Valid CSV with 3 records and 4 columns",
  "s3Key": "validated/csv/1234567890_data.csv",
  "s3Uri": "s3://your-bucket-name/validated/csv/1234567890_data.csv"
}

Error Response:

{
  "statusCode": 400,
  "valid": false,
  "message": "CSV file has no data records"
}

Validation Rules

CSV Validation

• File must not be empty
• Must have at least one data record (excluding header)
• All rows must have the same number of columns
• Proper CSV format (parseable by Apache Commons CSV)

JSON Validation

• File must not be empty
• Must be valid JSON (parseable by Jackson)
• Can be JSON object or array

S3 Upload Structure

Files are uploaded with the following pattern:

validated/{file-type}/{timestamp}_{original-filename}

Example: validated/csv/1698765432000_data.csv

AWS IAM Permissions Required

The Lambda execution role needs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

Invoking via AWS CLI

aws lambda invoke \
  --function-name FileValidatorFunction \
  --payload '{"fileName":"test.csv","content":"name,age\nJohn,30","fileType":"CSV","bucketName":"your-bucket"}' \
  --region us-east-1 \
  response.json

cat response.json

Extending the Project

Add New Validators

1. Create a new validator class implementing FileValidator
2. Add validation logic in the validate() method
3. Register in FileValidatorHandler constructor

Example:

public class XmlValidator implements FileValidator {
    @Override
    public ValidationResult validate(String content) {
        // Your validation logic
    }
    
    @Override
    public String getFileType() {
        return "XML";
    }
}

Change AWS Region

Update region in S3Service.java:

public S3Service() {
    this.s3Client = S3Client.builder()
        .region(Region.US_WEST_2)  // Change here
        .credentialsProvider(ProfileCredentialsProvider.create("default"))
        .build();
}

Troubleshooting

"Access Denied" error

• Verify AWS credentials in ~/.aws/credentials
• Check IAM role permissions for Lambda
• Ensure S3 bucket exists and has proper permissions

"Class not found" error

• Ensure Maven Shade plugin is packaging dependencies
• Verify handler path: com.awsproject.lambda.FileValidatorHandler::handleRequest

Local testing fails

• Check Java version (must be 11+)
• Verify AWS credentials are configured
• Ensure S3 bucket exists

Dependencies

• AWS SDK for Java v2 (S3)
• AWS Lambda Java Core & Events
• Jackson (JSON processing)
• Apache Commons CSV
• SLF4J (logging)

License

This is a sample project for educational purposes.