-
Notifications
You must be signed in to change notification settings - Fork 0
Repository Structure
High-level map of what's in this repo and how the pieces relate. For data-flow diagrams see Architecture; for the framework's API see Reusable Framework.
| Path | What it is |
|---|---|
beam_pipeline_toolkit/pipeline/ |
Generic Beam engine: parse → validate → aggregate → sink. No business logic. |
beam_pipeline_toolkit/controlplane/ |
Submit / stop / read-evidence for a real Dataflow job (direct REST calls + ADC). |
beam_pipeline_toolkit/safety/ |
LeakGuard (forbidden-key scanner) + assert_gates_open (cloud gate). |
beam_pipeline_toolkit/ecommerce/ |
A generic e-commerce reference implementation built on the engine above. |
demo/pipeline/ |
The actual Beam pipeline the dashboard runs — Cyberbiz-shaped, synthetic-only. |
demo/app/ |
FastAPI backend + dashboard services (scenario runner, live-demo orchestration, evidence reads). |
demo/shared/ |
Synthetic event generator, scenario definitions, the safety validator this demo uses. |
tests/toolkit/ |
Proves the generic engine is actually generic, using an unrelated "widget count" pipeline. |
tests/demo/ |
Covers the actual demo (171 tests total across both suites). |
raw JSON → ParseNormalize → Validate → OrderFilter → tag origin → Flatten → AggregateAndEmit
- ParseNormalize — decodes JSON; anything unparseable is dropped to a count-only tagged output, raw bytes never travel further.
- Validate — structural checks (required fields, shop-id whitelist, ISO date, non-negative counts). Failures emit a sanitized reason code, never the raw record.
- OrderFilter — the "DBT-compatible" business rule — a plausible stand-in, not a claim about a real dbt model.
-
AggregateAndEmit — keys by
(run_id, shop_id, metric_date), groups, and emits the x1–x6 metric row + telemetry timeline + invalid summary, each checked byassert_safe()before being yielded.
Beam techniques worth knowing: TaggedOutput for branching, Metrics.counter for the custom counters visible in the real Dataflow console, and a one-shot AfterProcessingTime trigger rather than a repeating streaming trigger — deliberate, since this publishes one bounded batch per demo run.
grep -rn "beam_pipeline_toolkit" demo/ → 0 results
grep -rn "from demo" beam_pipeline_toolkit/ → 0 results
beam_pipeline_toolkit/ and demo/ are two independent, parallel implementations of the same shape — not "demo built on top of the framework." The generic engine's build_pipeline() is structurally the same graph as demo/pipeline/main.py, except every business decision (what's valid, how to key records, what the outcome row looks like) is a caller-supplied parameter instead of hardcoded. beam_pipeline_toolkit/ecommerce/ is a separate, independently-named worked example — not the demo's code copy-pasted.
| Layer | Verdict | Why |
|---|---|---|
pipeline/ + controlplane/ + safety/ (toolkit) |
Reuse as-is | Zero business logic; already proven with an unrelated pipeline in tests/toolkit. |
ecommerce/ (toolkit) |
Reuse as a template | Right shape (sessions/orders/customers, Decimal money math, pluggable order policies) — swap field names/policies for the real schema. |
demo/pipeline/ |
Reuse the design, not the code | Hardcoded to synthetic-only data (a 4-value shop-id whitelist, a filter rule explicitly disclaimed as not matching real dbt logic). |
demo/app/ (dashboard) |
Reuse the pattern, rebuild the UI | "Everything through a leak guard before it reaches the browser" and "gate every mutating action" are the reusable parts; the specific tabs/scenarios are demo-shaped. |
To build one real pipeline: start from beam_pipeline_toolkit, using ecommerce/ as a starting point if the shape fits. Don't try to graduate demo/'s code straight into production — its safety constraints are deliberately synthetic-only.
Everything in this project's dashboard (scenario picker, metrics tables, explanation rail, live-demo control panel) is custom-built here (FastAPI + hand-written HTML/JS) — only the "Open native Dataflow job" link leaves this app, for Google's own console.
- Architecture — data flow diagrams
- Reusable Framework — how to build your own pipeline on the toolkit
- Safety Model — the leak guard and cloud gate in detail
- Dashboard Guide — how to run and use the dashboard