python-etl-pipeline

A complete, production-style ETL (Extract, Transform, Load) pipeline built in Python, with structured logging, configurable data sources, automated testing, Docker containerization, and a multi-stage GitHub Actions CI/CD pipeline.

Features

• Configurable extraction from a built-in sample dataset, CSV, or JSON files, controlled entirely via environment variables.
• Schema and null validation before transformation, with clear custom exceptions (ExtractionError, ValidationError, TransformationError, LoadError) for predictable error handling.
• Structured logging throughout every pipeline stage instead of raw print statements.
• Optional JSON output of pipeline results to disk.
• Comprehensive test suite (pytest) covering happy paths, edge cases, and failure modes, with coverage reporting.
• Dockerized with a multi-stage build, a non-root runtime user, and a container healthcheck.
• CI/CD via GitHub Actions: linting (black, isort, flake8, mypy), a Python version test matrix (3.10–3.12), and a Docker build/test stage with layer caching.

Project structure

python-etl-pipeline/
├── etl/
│   ├── config.py       # Environment-driven pipeline configuration
│   ├── exceptions.py   # Custom exception hierarchy
│   └── pipeline.py     # extract / transform / load / run_pipeline
├── tests/
│   ├── test_config.py
│   └── test_pipeline.py
├── .github/workflows/ci-cd.yml
├── Dockerfile
├── requirements.txt
├── requirements-dev.txt
├── pyproject.toml
└── setup.cfg

Getting started

Local setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt

Run the pipeline

python -m etl.pipeline

Configuration

All settings are optional and read from environment variables:

Variable	Default	Description
ETL_SOURCE_PATH	(none)	Path to a CSV/JSON source file
ETL_SOURCE_FORMAT	builtin	One of builtin, csv, json
ETL_OUTPUT_PATH	(none)	If set, writes pipeline results as JSON
ETL_LOG_LEVEL	INFO	Standard Python logging level
ETL_CHUNK_SIZE	1000	Reserved for future chunked processing

Example using a CSV source:

export ETL_SOURCE_PATH=./data/customers.csv
export ETL_SOURCE_FORMAT=csv
export ETL_OUTPUT_PATH=./output/result.json
python -m etl.pipeline

Run tests

export PYTHONPATH=.
pytest tests/ --cov=etl --cov-report=term-missing

Lint and format

black etl tests
isort etl tests
flake8 etl tests
mypy etl

Docker

docker build -t python-etl-pipeline .
docker run --rm python-etl-pipeline

License

Distributed under the MIT License. See LICENSE for details.