Warehouse

Warehouse Models

A dbt project (pipeline/dbt/ in the repository) turns the pipeline's stored rows into tested tables. The same models run on the lite-mode DuckDB file and on the Docker stack's PostgreSQL. The full page is docs/WAREHOUSE.md.

Not released yet: run it from a clone. dbt is a development dependency group, not part of the package.

uv sync --group dbt
uv run python scripts/build_lite_warehouse.py --db .prohori/pipeline/prohori.duckdb
uv run dbt build --project-dir pipeline/dbt --profiles-dir pipeline/dbt     # add --target postgres for the stack

What you get

Model	Answers
daily_attack_counts, daily_alerts_by_rule	What happened each day, and which rules fired
top_source_ips	Which addresses produced the most alerts and incidents
incident_durations	How long incidents last
rule_precision	How many of each rule's alerts were real attacks (simulated logs)
alert_latency, detection_latency	How long activity ran before a rule fired, and how long the pipeline took
attack_detection	Whether and how fast each simulated attack was detected
model_performance	Flow-model metrics per evaluation, and the change since the last one

Precision and time to detect need the simulator's labels (prohori pipeline load-labels LOG). Model results come from prohori pipeline load-results results/ml_*.json.

Checks

dbt build runs 68 data-quality checks: unique and non-null ids, accepted values, ranges, duplicates, cross-table consistency, no future timestamps, and a warning for dead letters. dbt source freshness warns when no new rows arrived for a day. On the simulator's eight scenarios, the models' rule precision equals the simulator's own evaluation alert for alert, which CI checks.