Controlplane Module EN

繁體中文版

Control Plane Module — beam_pipeline_toolkit/controlplane/

A generic Dataflow control-plane API: submit a job, request cancellation, read job evidence. All REST calls, all ADC-authenticated, never a gcloud subprocess.

google_auth_client.py

get_authorized_session() — deferred-import ADC (google.auth.default()) + AuthorizedSession. Raises CloudAccessError on failure. Every other module in this package gets its HTTP session from here.

dataflow_submit.py

• submit_job(*, gate_check, submit_fn) — runs gate_check() (raises on failure) then submit_fn(); wraps any failure as CloudSubmissionUnavailable so library internals never leak to the caller's API layer.
• generate_job_name(prefix) — <prefix>-<UTC timestamp>-<random hex suffix>, using only Dataflow-legal job-name characters. Optional convenience.

dataflow_stop.py

request_job_cancellation(*, project, job_id, region) — PUT to projects.locations.jobs.update with requestedState: JOB_STATE_CANCELLED. Returns the reported state right after the request (typically CANCELLING — cancellation is asynchronous).

dataflow_evidence.py

Read-only (GET) only — nothing here submits, cancels, drains, or updates a job.

• get_dataflow_job_summary(...) → DataflowJobSummary (name, short id, region, type, state, timestamps, console URL).
• get_dataflow_job_metrics(...) → list of DataflowCounterValue for each requested counter name; anything not found in the real response is found=False, never fabricated.
• is_job_running(...) → True only if state is RUNNING.
• extract_user_counter(...) — pulls one named user-namespace counter's scalar out of a raw .../metrics JSON response.
• job_url/console_url — REST/console URL builders.

Pair this with your own BigQuery-backed evidence lookups for a fuller operational picture — this module only knows what the Dataflow REST API itself reports.