File Guide EN

繁體中文版

File Guide

Every source file in the repo, what it does, and where to find it (main branch). Test files are listed on Testing.

Root

File	Purpose
README.md	Top-level intro, install/test instructions, repo layout
pyproject.toml	Project metadata + pytest config (testpaths = tests)
setup.py	Packaging file used only by DataflowRunner (--setup_file) to stage the package onto remote workers; not used by local DirectRunner
requirements.txt	Direct dependencies (apache-beam[gcp], google-auth, requests, pytest, build)
requirements.lock.txt	Fully pinned transitive dependency versions
.gitignore	Excludes .venv, caches, build artifacts, logs, .idea/.vscode

beam_pipeline_toolkit/

File	Purpose
README.md	Full API documentation and worked code samples for every package below
__init__.py	Package docstring only

pipeline/ — see Pipeline Module

File	Purpose
__init__.py	Package docstring
main.py	build_pipeline, run_local, run_cloud, apply_streaming_window — assembles and runs the full transform graph
parsing.py	ParseNormalize — PARSE_NORMALIZE stage: JSON decode + caller normalize_fn
validation.py	Validate — VALIDATE stage: applies caller validate_fn, tags valid/invalid
aggregation.py	AggregateAndEmit, tag_origin — AGGREGATE stage: key, group, call build_outcome_fn, leak-check every output
sinks.py	write_jsonl/read_jsonl (local), build_cloud_pubsub_sink/build_cloud_bigquery_sink (gated cloud sinks)
schema.py	PipelineSchema — the caller-supplied contract (metrics namespace, allowed stages/statuses)
options.py	BaseToolkitPipelineOptions — shared local/cloud/gate CLI flags, no hardcoded defaults
metrics.py	serialize_metric_row/decimal_to_json_string — Decimal-safe JSON serialization
telemetry.py	build_telemetry_event — sanitized, schema-validated telemetry event builder
run_manifest.py	build_run_manifest — sanitized summary of one pipeline run

controlplane/ — see Control Plane Module

File	Purpose
__init__.py	Package docstring
google_auth_client.py	get_authorized_session — shared ADC-only authenticated HTTP session
dataflow_submit.py	submit_job, generate_job_name — gate-check-then-submit pattern
dataflow_stop.py	request_job_cancellation — requests JOB_STATE_CANCELLED via REST
dataflow_evidence.py	get_dataflow_job_summary, get_dataflow_job_metrics, is_job_running — read-only job metadata + counters

log/ — see Log Module

File	Purpose
__init__.py	Package docstring
execution_log.py	exception_event, build_execution_log_event — sanitized exception/lifecycle events
stage_timing.py	StageTimer, build_stage_timing_event, summarize_stage_durations — per-stage wall-clock timing
dataflow_logs.py	get_dataflow_job_logs, get_all_dataflow_job_logs — read-only Cloud Logging reader for a job's own logs

safety/ — see Safety Module

File	Purpose
__init__.py	Package docstring
leak_guard.py	LeakGuard, assert_safe, is_safe — recursive forbidden-key scanner
cloud_gate.py	assert_gates_open — the single choke point before building any real cloud client

ecommerce/ — see Ecommerce Module

File	Purpose
__init__.py	Package docstring
schemas.py	SessionRecord/OrderRecord/CustomerRecord/DailyMetricRow/InvalidRecord NamedTuples
metrics.py	compute_daily_metrics — Decimal-safe conversion-rate/AOV/revenue math
order_policies.py	exclude_cancelled_and_returned, accept_all_orders, partition_by_policy — pluggable order-validity rules
validation.py	build_validate_fn, deduplicate_sessions — structural checks + duplicate-session handling
pipeline.py	build_ecommerce_pipeline, build_outcome_fn_factory — the complete worked example

tests/

Full list and descriptions on the Testing page.