Distributed Spark/Delta Lake ETL with automated data quality checks and live monitoring
Medallion architecture · null/duplicate/completeness checks · FastAPI control room · M+ records
A distributed Spark/Delta Lake ETL pipeline with automated data quality checks and a live monitoring dashboard, built for European data platform teams processing high-volume transaction data.
Live: https://flowlake-demo.vercel.app
All screenshots below are from the actual running stack, same light theme, same wave/lake logo, real numbers computed from the real UCI Online Retail dataset (541,909 rows). Click any thumbnail for the full-size image.
 Dashboard: Overview |
 Dashboard: Revenue Analytics |
 Dashboard: Run History & Quality Report |
 Dashboard: Customer Lookup |
 Real Spark Cluster: 2 alive workers, completed BatchETLJob runs submitted by Airflow
|
 Real Apache Airflow Scheduler: the DAG actually deployed and running |
FlowLake ingests raw transaction data, applies business transformations (deriving totals, date partitions, year/month columns), runs automated data quality checks (null percentages, duplicates, completeness, record counts), and writes validated output to a Delta-Lake-style data lake. A FastAPI control room dashboard lets you trigger pipeline runs, watch metrics update live, and inspect the latest data quality report.
European retailers, logistics companies, and banks process millions of transaction records daily and need automated, auditable ETL pipelines with built-in data quality gates, this project demonstrates exactly that pattern at small scale, ready to extend to a real Spark cluster.
| Technology | Version | Purpose |
|---|---|---|
| Apache Spark (Scala) | 3.5.0 | Distributed batch ETL jobs |
| PySpark | 3.5.0 | Python Spark job wrappers |
| Delta Lake | 3.0.0 | ACID data lake storage format |
| Apache Airflow | 2.7.3 | Pipeline orchestration / scheduling, deployed and running the real DAG daily |
| FastAPI | 0.110+ | Pipeline control room API |
| Pandas / NumPy | 2.x / 1.24+ | Lightweight ETL simulation for the demo API |
| Docker / Docker Compose | latest | Containerized Spark cluster + API + Airflow |
git clone https://github.com/Hamilas/flowlake.git
cd flowlake
cp .env.example .env
docker compose up -d apiOpen http://localhost:8025 for the pipeline control room dashboard, or http://localhost:8025/docs for the API reference.
To also start the Spark cluster (master + 2 workers) for real Spark-job execution:
docker compose up -dTo access the Airflow scheduler UI (runs the real DAG daily at 02:00, or trigger
it manually): http://localhost:8089, login admin / flowlake (set via
_AIRFLOW_WWW_USER_USERNAME / _AIRFLOW_WWW_USER_PASSWORD in docker-compose.yml).
- Extract -> Transform -> Validate -> Load pipeline (pandas-based API + Scala/PySpark jobs for real Spark clusters)
- Automated data quality checks: null %, duplicates, completeness, record count, value range
- Live control room dashboard with KPIs, run history, and quality reports
- Request logging middleware, Pydantic input validation, and structured HTTP error handling (404/422/500) on every endpoint
- Fully configurable via
.envandconfig/pipeline.yaml - Dockerized: API service + Spark master/worker cluster
- Airflow DAG for daily scheduled runs
/metricsand/datasetsendpoints for monitoring dashboards
Example pipeline runs captured from the live API:
| Run ID | Records | Quality Checks Passed | Duration |
|---|---|---|---|
| 832f95d5 | 2,010 | 15/15 (100%) | 0.104s |
| 0fbde1bc | 8,040 | 15/15 (100%) | 0.064s |
| 19bf6b74 | 12,060 | 15/15 (100%) | 0.104s |
- Retail/E-commerce (e.g., Zalando, OTTO): nightly batch processing of transaction logs with automated data quality gates before BI dashboards refresh
- Logistics (e.g., DHL, DPD): validating shipment/scan event data at scale before loading into analytics warehouses
- Banking/Fintech (e.g., N26, Deutsche Bank): auditable ETL pipelines with quality checks for regulatory reporting (completeness, null thresholds, duplicate detection)
Rayen Lassoued github.com/Hamilas | LinkedIn
MIT