beam-pipeline-toolkit

A standalone, reusable Apache Beam pipeline framework: generic parse -> validate -> aggregate -> sink wiring, a Dataflow control-plane API (submit / stop / read job evidence), an opt-in execution-log / stage-timing / Cloud-Logging layer, and a small safety layer (a recursive PII/secret leak guard, and a cloud-action confirmation gate) -- plus an optional, ready-to-adapt e-commerce domain extension.

This repo contains only the generic framework. There is no demo application, no dashboard, no one-off business logic, and no reference to any specific company or product -- see beam_pipeline_toolkit/README.md for the full API documentation, including the logging layer (beam_pipeline_toolkit.log) built for exactly this: being picked up and reused by a different pipeline later.

Install

python -m venv .venv
.venv\Scripts\activate        # Windows
pip install -r requirements.txt

Test

pytest

tests/ proves genericity with deliberately trivial, made-up business logic (a "widget count per warehouse" pipeline) plus the optional e-commerce domain layer, and exercises the opt-in logging layer against both.

Layout

beam_pipeline_toolkit/
  pipeline/       Generic Beam pipeline: parse, validate, aggregate, sinks, run manifest
  controlplane/   Generic Dataflow control-plane API: submit, stop, read job evidence
  log/            Execution-log events, stage-timing events, Dataflow/Cloud Logging reader
  safety/         Leak guard (forbidden-key scanner) + cloud-action gate
  ecommerce/      Optional e-commerce domain layer built on the above
tests/            Full test suite for everything above