Data Analysis & Programming (2026-1)

Academic project for the Data Analysis & Programming course. It implements a Medallion-style data architecture orchestrated with Apache Airflow to move data from MongoDB into analytics-ready layers.

Goal

Build a reproducible data pipeline with two active stages:

• Raw data ingestion into Bronze.
• Data cleaning and standardization into Silver (Parquet format).

Repository Structure

.
├── airflow/
│   ├── dags/
│   │   ├── bronze_ingestion_dag.py
│   │   ├── silver_processing_dag.py
│   │   └── gold_analysis_dag.py
│   ├── docker-compose.yaml
│   ├── Dockerfile
│   ├── requirements.txt
│   └── config/
├── dashboard/
│   ├── assets/
│   │   └── style.css
│   ├── governance_app.py
│   └── storytelling_app.py
├── datalake_bronze/
├── datalake_silver/
├── datalake_gold/
├── notebooks/
├── workshop_1/
├── workshop_2/
├── workshop_3/
├── workshop_4/
├── catchUp/
└── getRawDataService/

Data Architecture

• Bronze (datalake_bronze): raw JSON files.
• Silver (datalake_silver): cleaned and typed Parquet files.
• Gold (datalake_gold): reserved for curated aggregates and final consumption.

Current DAGs

• bronze_ingestion_webscraping

  • Schedule: daily.
  • Action: extracts the most recent document from the web scraping collection and writes it to Bronze.

• bronze_ingestion_twitter

  • Schedule: Monday and Thursday at 06:00 UTC.
  • Action: extracts Twitter/comments documents and writes it to Bronze.

• silver_processing_dag

  • Schedule: Monday and Thursday at 07:00 UTC.
  • Action: detects new Bronze JSON files, applies cleaning/transformation logic, and writes Parquet files to Silver.

Requirements

• Docker and Docker Compose.
• Git.
• Optional for local development outside containers: Python 3.10+.

Environment Configuration

The pipeline uses environment variables for MongoDB connectivity and data lake paths.

Define at least the following in airflow/.env:

MONGO_URI=<your_mongodb_uri>
MONGO_DB=<your_database_name>
BRONZE_BASE_PATH=/opt/airflow/datalake_bronze
SILVER_BASE_PATH=/opt/airflow/datalake_silver

Security note:

• Never commit real credentials to GitHub.
• Use placeholders in documentation and keep secrets only in local .env files or a secret manager.

Quick Start (Airflow with Docker)

From the airflow directory:

cd airflow
docker compose up airflow-init
docker compose up -d

Then open Airflow at http://localhost:8080.

To stop services:

docker compose down

Interactive Visualization Dashboards

The presentation layer consists of two independent Plotly Dash web applications located in dashboard/. They load the latest Parquet folders directly from the Gold data lake layer.

Both dashboards feature a dynamic path resolver, allowing them to run seamlessly without absolute path configurations. The path is determined automatically from the GOLD_PATH environment variable (if running in Docker) or from parent/relative directories containing datalake_gold (if running locally on a developer's laptop).

Running Dashboards in Docker (Included in Compose)

By default, launching docker compose up -d in the airflow/ directory starts both dashboards on their default ports:

• Governance Dashboard: http://localhost:8050
• Storytelling Dashboard: http://localhost:8051

If you modify dashboard files on the host, you must rebuild the containers to apply the changes:

cd airflow
docker compose down
docker compose up --build -d

Running Dashboards Locally (Outside Docker)

1. Set up a virtual environment and install dependencies:

   python -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt
   pip install dash

2. Stop the Docker dashboard services to release ports 8050 and 8051:

   cd airflow
   docker compose stop dashboard-governance dashboard-storytelling

3. Launch the applications locally:

   • Governance & Data Quality Dashboard (localhost:8050): Surfaces overall null ratios, duplicate rates, and schema compliance in visual metric cards colored by warning thresholds. Includes a dedicated Business Rules quality threshold card and detailed helper texts. Shows horizontal null profiles, text length distributions, volume over time, and outlier detection (IQR) with channel filters.

     python dashboard/governance_app.py

   • Business Storytelling & Virality Dashboard (localhost:8051): Synthesizes net sentiment, weekly plain-language executive summaries, brand mention shares, product line sentiment breakdowns, representative positive/critical comment cards, and social virality scatter plots. Explains VADER scoring metrics, and features a dynamic brand selector that automatically detects and populates new incoming brands from raw Parquet files on the fly.

     python dashboard/storytelling_app.py

Step-by-Step to Inspect MongoDB

• Open airflow/.env and confirm the MongoDB credentials are there.

• From the repository root, install the local dependencies if you want to run the explorer script:

python -m pip install pymongo python-dotenv

• Run the local explorer script:

python notebooks/explore_mongodb.py

• Read the output:

  • It shows the databases visible to your user.
  • It lists the collections inside dataProgrammingAnalysis.
  • It prints a few sample documents from each collection.

• If you want to see the raw files already generated by the pipeline, check:

  • datalake_bronze/
  • datalake_silver/

• If Airflow is not running yet, start it from airflow/ with:

cd airflow
docker compose up airflow-init
docker compose up -d

• Open Airflow at http://localhost:8080 and trigger the DAGs to populate Bronze and Silver.

Expected Data Flow

1. Ingestion DAGs write JSON files into datalake_bronze.
2. The Silver DAG reads the latest file per source.
3. Cleaning and type normalization are applied.
4. Processed Parquet files are written into datalake_silver.

Workshops and Supporting Material

• workshop_1: project definition, user stories, and data source validation.
• workshop_2: technical architecture, compose setups, and Bronze/Silver pipeline DAGs.
• workshop_3: PySpark analytical Gold layer workflows, quality KPIs, and storytelling specifications.
• workshop_4: interactive Plotly Dash presentation dashboards design, styling, and LaTeX report.
• notebooks: exploratory analysis notebooks and MongoDB inspectors.

Project Status

• Bronze layer: operational.
• Silver layer: operational.
• Gold layer: operational (PySpark analytical summaries generated).
• Presentation layer: operational (Interactive Dash dashboards active).

Authors

• Juan Diego Grajales Castillo
• Carmen Sofia Florez Juajibioy
• Edgar Alejandro Mora Chala

---

Last updated: 2026-04-12