Skip to content

Releases: adnanphp/openbi

Release list

OpenBI v3.0.0 — Cloud-Readiness Layer

OpenBI v3.0.0 — Cloud-Readiness Layer Latest
Latest

Choose a tag to compare

View all tags
@adnanphp adnanphp released this 07 Oct 21:35
v3.0.0
bb9dbc3

Summary

Extends OpenBI with the eight-layer cloud-readiness work planned in
docs/v3-roadmap.md. Everything runs locally at zero cost, but every
component speaks the same protocol or implements the same concept as its
managed AWS or GCP counterpart.

What's added

Phase Component Maps to
1 Traefik ingress ALB / Cloud LB
2 MinIO S3 / GCS
3 Redis cache-aside ElastiCache / Memorystore
4 LocalStack S3 + SQS S3 + SQS
5 OpenFaaS serverless Lambda / Cloud Functions
6 Terraform IaC (full) Terraform AWS/GCP providers
7 Loki + structured logging CloudWatch Logs / Cloud Logging
8 Cloud mapping doc —

Reproducibility

From an empty Docker host:

cd infrastructure/terraform
terraform init
terraform apply
./scripts/bootstrap-kind-images.sh
Assets 2
Loading

OpenBI v2.0 — Complete Modern Data Platform

OpenBI v2.0 — Complete Modern Data Platform

Choose a tag to compare

View all tags
@adnanphp adnanphp released this 30 Sep 21:07
v2.0.0
7bc8fc8

OpenBI v2.0

The complete modern data platform. Batch + streaming. Nine phases. 100% open source.

What's inside

Batch pipeline (Spark + Delta Lake)

  • PySpark ingestion → Delta Bronze (partitioned by category)
  • Star schema in Delta Silver (5 dims + 1 fact, partitioned by year)
  • Gold aggregations → Postgres warehouse_big
  • 1,000,000 rows processed end-to-end

ML layer (Spark MLlib)

  • RFM + KMeans segmentation (7 business segments)
  • Sales forecasting (baseline / ETS / XGBoost, MAPE-based selection)
  • Outputs published to Postgres, consumed by Superset

Real-time pipeline (Kafka + Structured Streaming)

  • Kafka 3.7 in KRaft mode (no ZooKeeper)
  • Spark Structured Streaming: Kafka → Delta Bronze
  • JDBC micro-batch sink: Delta → Postgres orders_realtime
  • 12,000+ events processed during the demo run

Analytics engineering (dbt)

  • 6 staging views, 3 mart tables
  • 49 data-quality tests
  • Auto-generated lineage docs

Orchestration (Airflow)

  • 3 DAGs: daily batch, weekly ML, hourly smoke test
  • Metadata in a dedicated airflow Postgres database

Observability (Prometheus + Grafana)

  • Custom exporter for data + ML metrics
  • 4 alert rules (data stale, rows dropping, MAPE high, scheduler down)
  • 5-panel Grafana dashboard

Serving layer

  • Apache Superset: 3 batch dashboards + 1 real-time dashboard
  • FastAPI: 8 endpoints with Swagger docs

CI/CD

  • ci.yml — v1 (pandas + Postgres)
  • ci-v2.yml — v2 (Spark + Delta)
  • Both green on every push

Stack

Apache Spark · PySpark · Spark SQL · Spark MLlib · Structured Streaming ·
Delta Lake · Kafka · Airflow · dbt · PostgreSQL · Apache Superset ·
FastAPI · Prometheus · Grafana · Docker Compose · pytest · GitHub Actions

Install & run

git clone https://github.com/adnanphp/openbi.git
cd openbi
pip install -r requirements.txt && pip install -e .

docker compose -f docker-compose.yml \
               -f docker-compose.bigdata.yml \
               -f docker-compose.kafka.yml \
               -f docker-compose.dbt.yml up -d

make bigdata       # batch: bronze → silver → gold → postgres
make bigdata-ml    # ML: RFM + forecasting
make dbt-build     # dbt: 9 models + 49 tests
make streaming-up  # Kafka + topic creation
Loading