NorthStar Commerce — Open-Source Data Stack

A complete, runnable end-to-end data stack for a fictional e-commerce company. Built entirely with open-source tools. No cloud accounts required.

Stack

Layer	Tool	Version
Warehouse	DuckDB	1.1.3
Transform	dbt-core + dbt-duckdb	1.8
Orchestration	Dagster	1.9
Data generation	Faker + Python	3.11+
Data quality	dbt schema tests + custom SQL

Quick Start

# 1. Clone / navigate to the project
cd northstar-data-stack

# 2. Create a virtual environment
python -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Run the entire pipeline
python run_pipeline.py

That's it. The pipeline will:

• Generate ~135k rows of realistic synthetic data across 6 tables
• Load them into a local DuckDB warehouse
• Run 11 dbt models (staging → intermediate → marts)
• Execute data quality tests
• Write a KPI summary to reports/summary.md

Project Structure

northstar-data-stack/
├── ingestion/
│   ├── generate_data.py     # Synthetic data generator (Faker)
│   └── load_raw.py          # CSV → DuckDB raw schema loader
│
├── data/raw/                # Generated CSV files (gitignored)
├── warehouse/               # DuckDB file (gitignored)
│
├── transform/               # dbt project
│   ├── models/
│   │   ├── staging/         # Views — type-cast, renamed, filtered
│   │   ├── intermediate/    # Ephemeral — business logic, joins
│   │   └── marts/
│   │       ├── finance/     # fct_orders, mart_daily_revenue
│   │       ├── marketing/   # dim_customers, mart_customer_cohorts
│   │       └── product/     # dim_products, mart_product_performance
│   ├── tests/               # Custom singular tests
│   └── macros/
│
├── orchestration/           # Dagster asset graph + schedules
│   └── northstar/
│       ├── assets.py        # Software-defined assets
│       ├── jobs.py          # full_pipeline_job, transform_only_job
│       └── schedules.py     # Daily 03:00 UTC schedule
│
├── docs/
│   ├── architecture.md      # Stack diagram + tool choices
│   ├── data_dictionary.md   # Column-level documentation
│   └── runbook.md           # Operations + troubleshooting
│
├── reports/                 # Auto-generated KPI summaries
├── run_pipeline.py          # One-shot runner (no Dagster needed)
└── requirements.txt

Querying the Warehouse

duckdb warehouse/northstar.duckdb

-- Daily revenue trend (last 30 days)
SELECT date, gross_revenue, revenue_7d_ma, revenue_wow_growth
FROM marts_finance.mart_daily_revenue
ORDER BY date DESC LIMIT 30;

-- Customer RFM segments
SELECT rfm_segment, COUNT(*) AS customers,
       AVG(lifetime_revenue) AS avg_ltv
FROM marts_marketing.dim_customers
GROUP BY 1 ORDER BY 2 DESC;

-- Top products by revenue
SELECT product_name, category, units_sold,
       total_revenue, velocity_tier
FROM marts_product.dim_products
ORDER BY total_revenue DESC LIMIT 20;

-- Cohort retention (Month 0 → Month 3)
SELECT cohort_month, months_since_acquisition,
       cohort_size, active_customers,
       ROUND(retention_rate * 100, 1) AS retention_pct
FROM marts_marketing.mart_customer_cohorts
WHERE months_since_acquisition BETWEEN 0 AND 3
ORDER BY cohort_month, months_since_acquisition;

dbt Docs

Generate and serve the full documentation site — every model, column, test, and source with a lineage graph:

cd transform
dbt docs generate --profiles-dir .
dbt docs serve --profiles-dir . --port 8080

Open http://localhost:8080. Three things worth clicking:

• Project tab — models organized by layer (staging / intermediate / marts), with column descriptions and test results
• Database tab — actual tables in DuckDB with row counts and column types
• Lineage graph — open any model, click the blue graph icon (bottom right) to see its full upstream/downstream DAG

Press Ctrl+C to stop the server.

Lineage in VS Code

Install the dbt Power User extension (innoverio.vscode-dbt-power-user). With the project settings already configured in .vscode/settings.json, open any model file and:

• Click Lineage in the panel at the bottom of the editor to see the model's DAG inline
• Hover over a {{ ref('...') }} to preview that model's columns
• Right-click any model in the file explorer → Run dbt model to execute it without leaving VS Code

Dagster UI

cd orchestration
DAGSTER_HOME=$(pwd) dagster dev -m northstar -p 3000

Open http://localhost:3000 to see the asset graph, trigger runs, and inspect metadata and logs for each asset.

Data Model

customers (5k)  ──┐
products  (200) ──┼──► orders (40k) ──► order_items (90k)
                  │
                  └──► web_events (80k)

campaigns (40)    (standalone)

dbt Lineage

raw.customers ──► stg_customers ──► int_customer_metrics ──► dim_customers
raw.products  ──► stg_products  ──► int_product_sales    ──► dim_products
raw.orders    ──► stg_orders    ──┐
raw.order_items ► stg_order_items ┴► int_orders_enriched ──► fct_orders
                                                          ──► mart_daily_revenue
                                                          ──► mart_customer_cohorts
                                                          ──► mart_product_performance
raw.campaigns ──► stg_campaigns   (standalone)
raw.web_events ─► stg_web_events  (standalone)

Data Quality Tests

• 55+ schema tests: unique, not_null, accepted_values, relationships
• 2 custom singular tests: revenue positivity, margin range
• dbt-utils: numeric range expressions on price/margin columns

Docs

• Architecture — full stack diagram, tool rationale
• Data Dictionary — every column documented
• Runbook — operations, common queries, troubleshooting