☁️ Cloud File Processing Pipeline

A production-ready FastAPI service that demonstrates a cloud-native file processing pipeline with Infrastructure as Code using Terraform.

Overview

This project provides a lightweight backend service for receiving processing requests while provisioning cloud infrastructure using Terraform. It demonstrates modern DevOps practices including containerization, infrastructure automation, testing, and CI.

Features

• FastAPI REST API
• Cloud Infrastructure with Terraform
• Docker containerization
• Docker Compose support
• Automated testing with Pytest
• GitHub Actions CI
• Infrastructure as Code (IaC)
• Production-ready project structure

---

Technology Stack

• Python 3.12
• FastAPI
• Terraform
• Docker
• Docker Compose
• Pytest
• GitHub Actions

---

Project Structure

cloud-file-processing-pipeline/
│
├── app/
│   ├── __init__.py
│   └── main.py
│
├── terraform/
│   └── main.tf
│
├── tests/
│   └── test_app.py
│
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── README.md
└── LICENSE

---

Installation

Clone the repository

git clone https://github.com/gerald1124-code/cloud-file-processing-pipeline.git
cd cloud-file-processing-pipeline

Create a virtual environment

python -m venv .venv

Activate

Windows

.venv\Scripts\activate

Linux/macOS

source .venv/bin/activate

Install dependencies

pip install -r requirements.txt

---

Run the API

uvicorn app.main:app --reload

Open

http://localhost:8000

Swagger UI

http://localhost:8000/docs

---

Docker

Build

docker build -t cloud-file-processing-pipeline .

Run

docker-compose up --build

---

Infrastructure

Initialize Terraform

cd terraform

terraform init
terraform validate
terraform plan
terraform apply

---

Testing

pytest

---

Continuous Integration

GitHub Actions automatically

• Install dependencies
• Run tests
• Validate the project
• Verify build

---

Future Improvements

• AWS S3 integration
• Azure Blob Storage
• Google Cloud Storage
• Background workers
• Celery
• Redis queue
• File metadata database
• Authentication
• Monitoring
• Logging
• Prometheus metrics

---

License

MIT License