AWS Data Engineering Pipeline

A complete data pipeline built on AWS for ingesting, cataloging, and cleansing customer data with quality issues.

Architecture

Raw Data (CSV) → S3 Input Bucket → Glue Crawler → Glue Data Catalog 
→ Glue ETL Job → S3 Output Bucket (Cleaned Data)

Components

• S3 Buckets: Storage for raw and cleaned data
• AWS Glue Crawler: Catalogs data schema automatically
• AWS Glue Database: Stores metadata catalog
• AWS Glue ETL Job: PySpark job for data cleansing
• IAM Role: Permissions for Glue to access S3

Features

Data Quality Issues Handled

• ✅ Duplicate record removal
• ✅ Missing value handling
• ✅ Invalid data correction (negative ages, unrealistic values)
• ✅ Text standardization (consistent formatting)
• ✅ Single CSV file output

Data Transformations

• Remove duplicate rows
• Filter rows with critical missing fields
• Fill missing values with defaults
• Validate and fix age ranges (0-120)
• Standardize text formatting (proper case, lowercase emails)
• Sort by customer_id
• Output as single CSV file

Files

• AWS_DATA_PIPELINE_DOCUMENTATION.md - Complete documentation
• glue_data_cleansing_job.py - Glue ETL job script
• sample_customer_data.csv - Sample data with quality issues
• glue-trust-policy.json - IAM trust policy for Glue
• s3-access-policy.json - S3 access policy for Glue role

Quick Start

Prerequisites

• AWS CLI configured
• AWS account with appropriate permissions
• Python 3.x (for local script editing)

Setup

1. Clone the repository

   git clone https://github.com/futurexskill/aws-copilot.git
   cd aws-copilot

2. Create S3 buckets

   aws s3 mb s3://your-raw-data-bucket --region us-east-1
   aws s3 mb s3://your-clean-data-bucket --region us-east-1

3. Create IAM role

   aws iam create-role --role-name YourGlueRole --assume-role-policy-document file://glue-trust-policy.json
   aws iam attach-role-policy --role-name YourGlueRole --policy-arn arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole
   aws iam put-role-policy --role-name YourGlueRole --policy-name S3Access --policy-document file://s3-access-policy.json

4. Upload data and script

   aws s3 cp sample_customer_data.csv s3://your-raw-data-bucket/raw/
   aws s3 cp glue_data_cleansing_job.py s3://your-raw-data-bucket/scripts/

5. Create Glue database

   aws glue create-database --database-input '{"Name": "your_db", "Description": "Your database"}'

6. Create and run Glue crawler

   aws glue create-crawler --name your-crawler --role YourGlueRole --database-name your_db --targets '{"S3Targets": [{"Path": "s3://your-raw-data-bucket/raw/"}]}'
   aws glue start-crawler --name your-crawler

7. Create and run Glue ETL job

   aws glue create-job --name your-etl-job --role arn:aws:iam::YOUR_ACCOUNT:role/YourGlueRole --command '{"Name": "glueetl", "ScriptLocation": "s3://your-raw-data-bucket/scripts/glue_data_cleansing_job.py", "PythonVersion": "3"}' --glue-version "4.0" --number-of-workers 2 --worker-type "G.1X"
   aws glue start-job-run --job-name your-etl-job

Results

• Original Records: 20 data rows (with duplicates and quality issues)
• Cleaned Records: 17 data rows (duplicates removed, data standardized)
• Output: Single CSV file in cleaned data bucket

Documentation

See AWS_DATA_PIPELINE_DOCUMENTATION.md for complete documentation including:

• Detailed architecture
• All AWS resource configurations
• Data quality issues and solutions
• Step-by-step instructions
• Troubleshooting guide
• Cost estimates
• Future enhancements

Cost Estimate

Estimated monthly cost for occasional runs: < $5/month

• S3 Storage: ~$0.50
• Glue Crawler: ~$0.44 per run
• Glue ETL Job: ~$0.88 per run

License

This project is created for educational and demonstration purposes.

Author

futurex-user
AWS Account: 522723766177
Created: December 20, 2025