Skip to content
BranchesTags

Repository files navigation

🏢 E-Commerce Data Engineering Pipeline

A production-ready end-to-end data engineering solution that extracts e-commerce data, transforms it through a modern data stack, and serves actionable insights via an interactive analytics dashboard.

Python Docker Apache Airflow PostgreSQL dbt Streamlit

📋 Table of Contents

🎯 Overview

This project demonstrates a complete data engineering workflow for an e-commerce platform. It showcases best practices in data extraction, transformation, loading (ETL), and analytics using industry-standard tools. The pipeline automatically fetches data from external APIs, transforms it using dbt, and serves insights through an interactive Streamlit dashboard—all orchestrated with Apache Airflow and containerized with Docker.

Key Objectives

  • Data Ingestion: Seamlessly extract e-commerce data from APIs
  • Data Transformation: Clean, validate, and aggregate data for analytics
  • Data Storage: Maintain a scalable, normalized PostgreSQL database
  • Data Visualization: Present actionable insights through interactive dashboards

✨ Features

📊 Data Pipeline

  • Automated Data Extraction: Fetches users, products, and shopping carts from DummyJSON API
  • Intelligent Scheduling: Apache Airflow manages workflow orchestration and monitoring
  • Error Handling & Logging: Comprehensive error tracking and retry mechanisms
  • Data Validation: Built-in data quality checks at each pipeline stage

📈 Analytics Dashboard

  • Products Analytics: Explore inventory with category, price, and rating filters
  • Customer Insights: Demographic analysis and geographic distribution
  • Sales Performance: Revenue trends, top performers, and KPI tracking
  • AI-Powered Recommendations: Machine learning-based product suggestions
  • Interactive Visualizations: Real-time charts and metrics powered by Streamlit

🔄 Data Transformation

  • dbt Models: Modular, version-controlled transformation logic
  • Staging Models: Raw data normalization and standardization
  • Mart Models: Business-ready aggregated tables
  • ML Features: Engineered features for machine learning pipelines

🐳 Infrastructure

  • Docker Containerization: Fully containerized for consistency and portability
  • Multi-Service Orchestration: Docker Compose manages all services
  • Scalable Architecture: Designed for easy scaling and customization

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Data Sources                             │
│              (DummyJSON API, External APIs)                  │
└────────────────────────┬────────────────────────────────────┘
                         │
┌────────────────────────▼────────────────────────────────────┐
│              Apache Airflow (Orchestration)                  │
│  ┌──────────────┬──────────────┬──────────────┐             │
│  │Fetch Users   │Fetch Products│Fetch Carts   │             │
│  └──────────────┴──────────────┴──────────────┘             │
└────────────────────────┬────────────────────────────────────┘
                         │
┌────────────────────────▼────────────────────────────────────┐
│              PostgreSQL (Data Warehouse)                     │
│  ┌────────────────────────────────────┐                     │
│  │  Raw Schema (Source Data)          │                     │
│  │  - users_raw, products_raw, ...    │                     │
│  └────────────────────────────────────┘                     │
└────────────────────────┬────────────────────────────────────┘
                         │
┌────────────────────────▼────────────────────────────────────┐
│                dbt (Transformation)                          │
│  ┌──────────────┬──────────────┬──────────────┐             │
│  │  Staging     │    Marts     │  ML Features │             │
│  │  Models      │    Models    │   Models     │             │
│  └──────────────┴──────────────┴──────────────┘             │
└────────────────────────┬────────────────────────────────────┘
                         │
┌────────────────────────▼────────────────────────────────────┐
│           Streamlit Analytics Dashboard                      │
│  ┌──────────┬──────────┬──────────┬──────────┐              │
│  │Products  │Customers │  Sales   │Analytics │              │
│  └──────────┴──────────┴──────────┴──────────┘              │
└─────────────────────────────────────────────────────────────┘

🛠️ Tech Stack

Component Technology Purpose
Orchestration Apache Airflow Workflow scheduling and monitoring
Data Extraction Python API integration and data extraction
Data Storage PostgreSQL Primary data warehouse
Data Transformation dbt (Data Build Tool) SQL-based transformation and testing
ML Pipeline Python (scikit-learn) Model training and recommendations
Analytics Streamlit Interactive web dashboard
API Server FastAPI REST API for data access
Containerization Docker & Docker Compose Infrastructure orchestration

📦 Prerequisites

Before getting started, ensure you have the following installed:

  • Docker Desktop (v20.10+)
    • Must be running during project startup
    • Allocate at least 4GB RAM for smooth operation
  • Git (v2.0+)
  • pgAdmin 4 (Optional, for database inspection)

System Requirements:

  • RAM: Minimum 4GB (8GB recommended)
  • Disk Space: At least 10GB free
  • OS: Windows, macOS, or Linux

🚀 Quick Start

Step 1: Clone the Repository

git clone https://github.com/yourusername/ecommerce-data-pipeline.git
cd ecommerce-data-pipeline/data-eng-project

Step 2: Start All Services

docker-compose up -d

Docker will automatically:

  • Download required images
  • Build custom containers
  • Start all services (Airflow, PostgreSQL, API, Dashboard)
  • Initialize databases

⏳ Wait 1–2 minutes for services to become healthy. Check status:

docker-compose ps

Expected output:

NAME                 STATUS
airflow-webserver    Up (healthy)
airflow-scheduler    Up (healthy)
airflow-worker       Up (healthy)
postgres             Up (healthy)
pgadmin              Up (healthy)
api-server           Up (healthy)

Step 3: Run Local Dashboard for Web interface

cd cd .\dashboard\

⏳ Create Vertual Environment

python -m venv venv

⏳ Activate Environment

# On Windows:
venv\Scripts\activate

# On macOS/Linux:
source venv/bin/activate

⏳ Install all Library that require.

pip install -r requirements.txt
streamlit run app.py

Step 4: Access Services

Service URL Credentials
Airflow UI http://localhost:8080 airflow / airflow
Streamlit Dashboard http://localhost:8501 None required
FastAPI Docs http://localhost:8000/docs None required
pgAdmin http://localhost:5050 admin@admin.com / admin

Step 4: Trigger the Pipeline

  1. Navigate to http://localhost:8080 (Airflow)
  2. Login with airflow / airflow
  3. Locate the ecommerce_ingest_pipeline DAG
  4. Click the ▶ Trigger DAG button
  5. Monitor execution in the Graph view

Step 5: View Your Data

Once the pipeline completes:

  1. Open http://localhost:8501 in your browser
  2. Explore the interactive dashboard:
    • 📦 Products: Category, price, and rating analytics
    • 👥 Customers: Demographics and location insights
    • 💰 Sales: Revenue trends and key metrics
    • 🔮 Recommendations: ML-powered product suggestions

📁 Project Structure

data-eng-project/
├── dags/                          # Airflow DAGs (orchestration logic)
│   ├── main_ingest.py            # Main pipeline DAG
│   ├── fetch_users.py            # User extraction task
│   ├── fetch_products.py         # Product extraction task
│   ├── fetch_carts.py            # Shopping cart extraction task
│   └── load_to_postgres.py       # Database loading task
│
├── ecommerce_dbt/                 # dbt project (transformation logic)
│   ├── models/
│   │   ├── staging/              # Raw data cleaning
│   │   ├── marts/                # Business-ready tables
│   │   ├── recommendations/      # ML feature engineering
│   │   └── ml_features/          # ML model inputs
│   ├── dbt_project.yml           # dbt configuration
│   └── profiles.yml              # Database connection config
│
├── dashboard/                     # Streamlit analytics dashboard
│   ├── app.py                    # Main dashboard application
│   ├── pages/                    # Multi-page analytics
│   │   ├── 01_products.py       # Product analytics page
│   │   ├── 02_sales.py          # Sales performance page
│   │   ├── 03_users.py          # Customer insights page
│   │   ├── 04_insights.py       # Advanced analytics
│   │   └── 05_recommendations.py # ML recommendations
│   ├── utils/                    # Helper functions
│   └── assets/                   # Static files (logos, data)
|   └── requirements.txt          # Requirement library 
│
├── api/                           # FastAPI backend
│   ├── app/
│   │   ├── main.py              # API entry point
│   │   ├── database.py          # Database connection
│   │   ├── models.py            # Pydantic models
│   │   ├── schemas.py           # API request/response schemas
│   │   └── routers/             # API endpoint routes
│   └── requirements.txt          # Python dependencies
│
├── ml_pipeline/                   # Machine learning service
│   ├── train_model.py           # Model training script
│   ├── load_ml_rules.py         # Load trained models
│   ├── requirements.txt          # ML dependencies
│   └── Dockerfile               # ML service container
│
├── postgres/
│   └── init/                    # PostgreSQL initialization scripts
│
├── config/
│   └── airflow.cfg              # Airflow configuration
│
├── docker-compose.yaml           # Multi-container orchestration
├── .env                          # Environment variables
└── README.md                     # This file

📖 Usage Guide

Running the Pipeline Manually

# Trigger the pipeline via CLI
docker exec airflow-scheduler airflow dags trigger ecommerce_ingest_pipeline

# View DAG status
docker exec airflow-scheduler airflow dags list

Accessing the Database

Option 1: pgAdmin GUI

  1. Open http://localhost:5050
  2. Register a new server:
    • Host: postgres
    • Port: 5432
    • Database: ecommerce_db
    • Username: admin
    • Password: admin123

Option 2: Command Line

# Connect to PostgreSQL directly
docker exec -it postgres psql -U admin -d ecommerce_db

# View tables in raw schema
SELECT table_name FROM information_schema.tables 
WHERE table_schema = 'raw';

# Query sample data
SELECT * FROM raw.users LIMIT 10;

Viewing Logs

# Airflow scheduler logs
docker-compose logs -f airflow-scheduler

# Airflow worker logs
docker-compose logs -f airflow-worker

# PostgreSQL logs
docker-compose logs -f postgres

# Dashboard logs
docker-compose logs -f dashboard

Stopping Services

# Gracefully stop all services (preserves data)
docker-compose down

# Stop and remove all data (full reset)
docker-compose down -v

# Stop specific service
docker-compose stop airflow-scheduler

💻 Development

Adding a New Data Source

  1. Create extraction task in dags/fetch_[source].py
  2. Add task to main DAG in dags/main_ingest.py
  3. Create dbt models in ecommerce_dbt/models/staging/
  4. Rebuild containers: docker-compose up -d --build

Modifying Transformations

# Test dbt models
docker exec ecommerce_dbt dbt test

# Run specific model
docker exec ecommerce_dbt dbt run --select staging.users

# Generate documentation
docker exec ecommerce_dbt dbt docs generate

Adding Dashboard Pages

Create new files in dashboard/pages/ following the naming convention NN_page_name.py.

Example template:

import streamlit as st
import pandas as pd
from utils.api import get_data

st.set_page_config(page_title="New Page", layout="wide")
st.title("📊 New Analytics Page")

# Your content here

🐛 Troubleshooting

Services Won't Start

Problem: docker-compose up fails or services are unhealthy

Solutions:

# Check Docker is running
docker ps

# View detailed error logs
docker-compose logs

# Rebuild containers
docker-compose up -d --build

# Complete restart
docker-compose down -v
docker-compose up -d

Airflow Not Accessible

Problem: Cannot connect to http://localhost:8080

Solutions:

# Wait for container to be fully healthy
docker-compose ps  # Check STATUS

# Restart Airflow
docker-compose restart airflow-webserver

# Check port conflicts
netstat -an | findstr 8080  # Windows
lsof -i :8080              # Mac/Linux

Pipeline Fails

Problem: DAG execution fails

Steps:

  1. Check Airflow logs: http://localhost:8080 → DAG → Latest Run → Logs
  2. Verify database connectivity
  3. Check API availability (DummyJSON API status)

Database Connection Issues

Problem: Cannot connect to PostgreSQL

Verify credentials:

docker exec -it postgres psql -U admin -d ecommerce_db

Common errors:

  • Port 5434 already in use → Change in docker-compose.yaml
  • Connection refused → Wait for container health check

Dashboard Errors

Problem: Streamlit dashboard shows errors

# Restart dashboard
docker-compose restart dashboard

# View logs
docker-compose logs dashboard

# Clear cache
docker exec dashboard streamlit cache clear

🤝 Contributing

Contributions are welcome! Please follow these guidelines:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

Code Standards

  • Follow PEP 8 for Python code
  • Add docstrings to functions and classes
  • Include unit tests for new features
  • Update documentation as needed

📝 License

This project is licensed under the MIT License — see the LICENSE file for details.

📞 Support

For issues, questions, or suggestions:

🎓 Educational Resources

This project demonstrates key data engineering concepts:

  • ETL Pipelines: Extract, Transform, Load workflows
  • Data Modeling: Dimensional modeling and normalization
  • Orchestration: Workflow scheduling with Apache Airflow
  • Data Quality: Testing and validation frameworks
  • Analytics: Real-time dashboards and visualization
  • DevOps: Containerization and infrastructure-as-code

🙏 Acknowledgments

Last Updated: May 2026 | Made with ❤️ for the Data Engineering Community

Restart a single service

docker-compose restart airflow-scheduler

About

No description, website, or topics provided.

Resources

Readme

Code of conduct

Code of conduct

Contributing

Contributing
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages