CSV to JSON Converter Lambda

AWS Lambda function that retrieves a CSV file from an S3 bucket, converts it to JSON format, and uploads it to another S3 bucket.

Architecture

Features

• Retrieves CSV files from S3
• Converts CSV data to JSON format
• Uploads JSON files to a destination S3 bucket
• Error handling and logging

Prerequisites

• Node.js 18.x or later
• AWS Account with S3 buckets configured
• AWS CLI configured (for deployment)

Event Structure

The Lambda function expects the following event structure:

{
  "sourceBucket": "my-source-bucket",
  "sourceKey": "data/input.csv",
  "destinationBucket": "my-destination-bucket",
  "destinationKey": "data/output.json"
}

• sourceBucket (required): Source S3 bucket name
• sourceKey (required): Path to the CSV file in the source bucket
• destinationBucket (required): Destination S3 bucket name
• destinationKey (optional): Path for the JSON file in the destination bucket. Defaults to the source key with .json extension

IAM Permissions

The Lambda execution role requires the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::my-source-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-destination-bucket/*"
    }
  ]
}

CSV Format

The function expects CSV files with:

• First row containing headers
• Comma-separated values
• One record per line

Example CSV:

name,age,city
John,30,New York
Jane,25,Los Angeles

Resulting JSON:

[
  {
    "name": "John",
    "age": "30",
    "city": "New York"
  },
  {
    "name": "Jane",
    "age": "25",
    "city": "Los Angeles"
  }
]

Deployment

1. Package the function:

zip -r function.zip index.js node_modules

2. Deploy using AWS CLI:

aws lambda create-function \
  --function-name csv-to-json-converter \
  --runtime nodejs18.x \
  --role arn:aws:iam::ACCOUNT_ID:role/lambda-execution-role \
  --handler index.handler \
  --zip-file fileb://function.zip

Testing

Invoke the function with a test event:

aws lambda invoke \
  --function-name csv-to-json-converter \
  --payload '{"sourceBucket":"my-source-bucket","sourceKey":"data/input.csv","destinationBucket":"my-destination-bucket"}' \
  response.json

Error Handling

The function returns:

• 200 status code on success
• 500 status code on error with error details in the response body

License

MIT