Big Data ELT (Spark + MinIO + Prefect)

Prerequisites

• Python 3.14 (project venv)
• Java 17 (required by Spark 3.5)
• Docker + Docker Compose

Setup

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Start services

docker compose up -d minio minio-setup postgres prefect-server spark-master spark-worker-1 spark-worker-2

Generate sample data

python script/generate_data.py

Run the ELT flows

python flows/bronze_ingestion.py
export SPARK_MASTER_URL="local[*]"
python flows/silver_ingestion.py
python flows/gold_ingestion.py

To use the Docker Spark cluster instead of local Spark, set: export SPARK_MASTER_URL="spark://localhost:7077"

Benchmark pandas vs PySpark

export JAVA_HOME=$(/usr/libexec/java_home -v 17)
export SPARK_MASTER_URL="local[*]"
python script/benchmark_processing.py

Visualize results with Streamlit

streamlit run streamlit_app.py

The dashboard reads the gold layer in MinIO (kpis.csv, ca_par_pays.csv, volumes_par_periode.csv, croissance_mensuelle.csv, distribution_montant.csv). Make sure the gold flow ran successfully before launching Streamlit.