Steam Reviews Analysis

End-to-end data pipeline that ingests Steam games and reviews to derive statistics from them (e.g. sentiment anaylsis, trends over time, comparisons between games). A personal project, meant to work both as a practical tool and as a technical showcase of a modern pipeline: API ingestion → Postgres warehouse → dbt transformation, all orchestrated and tested automatically by Dagster.

Repo structure

orchestration/     # Dagster code: assets, resources (Postgres/Steam/IGDB), jobs, schedules
dbt/                # dbt project: sources, staging models (marts coming soon)
db/init.sql         # DDL for the raw schema, run on Postgres' first startup
deploy/             # Dagster config (dagster.yaml, workspace.yaml)

Pipeline

flowchart LR
    IGDB[IGDB API\ndata dumps] -->|asset igdb_games| RAW1[(raw.igdb_games)]
    RAW1 --> CENSUS[asset steam_review_counts\nquery_summary census]
    CENSUS --> RAW2[(raw.steam_review_counts)]
    RAW2 --> BACKFILL[asset steam_reviews_backfill\ncursor pagination]
    BACKFILL --> RAW3[(raw.steam_reviews)]
    RAW3 -->|dbt staging| STG[stg_steam_review]
    STG -.mart in progress.-> MART[(marts)]

Loading

1. igdb_games — downloads the IGDB data dumps (games, external_games), keeps only games linked to a steam_app_id, and upserts them into raw.igdb_games.
2. steam_review_counts — for each game, fetches the Steam summary (query_summary: total reviews, score...).
3. steam_reviews_backfill — paginates the Steam API (appreviews) and loads the full payload of every review into raw.steam_reviews, upserting a row only if the review is more recent.
4. dbt (staging) — stg_steam_review flattens and types the raw review JSON (casts, renaming, etc.).
5. dbt (marts) — in progress.

Stack

• Dagster for orchestration (assets, resources, daily schedule).
• dbt for SQL transformations (staging → marts), linted with SQLFluff.
• PostgreSQL as the warehouse.
• Docker Compose to run the whole stack (Postgres, Dagster code, webserver, daemon).
• uv for Python dependency management, ruff + pre-commit for linting.

Running the project

Copy and edit .env.example. You'll need a valid IGDB API key to fetch games dumps:

cp .env.example .env   
docker compose up -d

The Dagster webserver is served on http://localhost:3001. The raw schema is created on Postgres' first startup (db/init.sql).

For local development (without Docker for the Dagster code):

uv sync --group dev
uv run dg dev         
uv run dbt run --project-dir dbt --profiles-dir dbt

Progress

• IGDB ingestion (game list + Steam mapping)
• Steam census (summaries) + full review backfill
• dbt staging (cleaned, typed reviews) with source tests
• CI (ruff, sqlfluff)
• dbt marts (actionable metrics) — in progress
• NLP tokenization asset (per-language, routed via the language field: powers word clouds / text-based marts for steam-reviews-website)
• Python tests on business logic (resources, casts)
• dbt build/dbt test run in CI
• Dagster alerting