Serverless Data Engineering Pipeline via AWS and Hugging Face Transformers library for sentiment analysis
This project demonstrates a serverless data engineering pipeline that processes and analyzes text data using AWS services and the Hugging Face Transformers library. The pipeline is designed to be scalable, cost-effective, and easy to maintain.
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Input S3 │────▶│ Lambda │────▶│ Output S3 │
│ Bucket │ │ Function │ │ Bucket │
└─────────────┘ └──────┬──────┘ └─────────────┘
│
▼
┌─────────────┐
│ Glue │
│ (ETL/ETL) │
└──────┬──────┘
│
▼
┌─────────────┐
│ CloudWatch │
│ (Monitor) │
└─────────────┘
- Serverless function for text processing
- Integrates with Hugging Face for sentiment analysis
- Triggered by new data uploads
- Input bucket for raw text data
- Output bucket for processed results
- Secure and scalable storage
- ETL operations
- Schema inference
- Data catalog management
- Pipeline monitoring
- Scheduled executions
- Performance metrics
-
Prerequisites
- AWS Account
- Python 3.8+
- AWS CLI configured
-
Installation
pip install -r requirements.txt
-
Configuration
- Set up AWS credentials
- Configure S3 buckets
- Deploy Lambda function
-
Running the Pipeline
python main.py
.
├── lambda_function.py # AWS Lambda handler
├── main.py # Main pipeline script
├── requirements.txt # Python dependencies
├── tf-requirements.txt # Terraform dependencies
└── utils/ # Utility functions
- Building Cloud Computing Solutions at Scale
- Python, Bash and SQL for Data Engineering
- Practical MLOps
This project is licensed under the MIT License - see the LICENSE file for details.