Skip to content

OpenBI v2.0 — Complete Modern Data Platform

Latest
Latest

Choose a tag to compare

View all tags
@adnanphp adnanphp released this 30 Sep 21:07
· 11 commits to main since this release
v2.0.0
7bc8fc8

OpenBI v2.0

The complete modern data platform. Batch + streaming. Nine phases. 100% open source.

What's inside

Batch pipeline (Spark + Delta Lake)

  • PySpark ingestion → Delta Bronze (partitioned by category)
  • Star schema in Delta Silver (5 dims + 1 fact, partitioned by year)
  • Gold aggregations → Postgres warehouse_big
  • 1,000,000 rows processed end-to-end

ML layer (Spark MLlib)

  • RFM + KMeans segmentation (7 business segments)
  • Sales forecasting (baseline / ETS / XGBoost, MAPE-based selection)
  • Outputs published to Postgres, consumed by Superset

Real-time pipeline (Kafka + Structured Streaming)

  • Kafka 3.7 in KRaft mode (no ZooKeeper)
  • Spark Structured Streaming: Kafka → Delta Bronze
  • JDBC micro-batch sink: Delta → Postgres orders_realtime
  • 12,000+ events processed during the demo run

Analytics engineering (dbt)

  • 6 staging views, 3 mart tables
  • 49 data-quality tests
  • Auto-generated lineage docs

Orchestration (Airflow)

  • 3 DAGs: daily batch, weekly ML, hourly smoke test
  • Metadata in a dedicated airflow Postgres database

Observability (Prometheus + Grafana)

  • Custom exporter for data + ML metrics
  • 4 alert rules (data stale, rows dropping, MAPE high, scheduler down)
  • 5-panel Grafana dashboard

Serving layer

  • Apache Superset: 3 batch dashboards + 1 real-time dashboard
  • FastAPI: 8 endpoints with Swagger docs

CI/CD

  • ci.yml — v1 (pandas + Postgres)
  • ci-v2.yml — v2 (Spark + Delta)
  • Both green on every push

Stack

Apache Spark · PySpark · Spark SQL · Spark MLlib · Structured Streaming ·
Delta Lake · Kafka · Airflow · dbt · PostgreSQL · Apache Superset ·
FastAPI · Prometheus · Grafana · Docker Compose · pytest · GitHub Actions

Install & run

git clone https://github.com/adnanphp/openbi.git
cd openbi
pip install -r requirements.txt && pip install -e .

docker compose -f docker-compose.yml \
               -f docker-compose.bigdata.yml \
               -f docker-compose.kafka.yml \
               -f docker-compose.dbt.yml up -d

make bigdata       # batch: bronze → silver → gold → postgres
make bigdata-ml    # ML: RFM + forecasting
make dbt-build     # dbt: 9 models + 49 tests
make streaming-up  # Kafka + topic creation
Assets 2
Loading